16:00:04 <jgriffith> #startmeeting cinder
16:00:05 <openstack> Meeting started Wed May 28 16:00:04 2014 UTC and is due to finish in 60 minutes.  The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:08 <jgriffith> hey everyone
16:00:09 <openstack> The meeting name has been set to 'cinder'
16:00:25 <bswartz> good afternoon
16:00:32 <jungleboyj> hey jgriffith
16:00:32 <DuncanT> hey
16:00:39 <xyang__> hi
16:00:45 <jgriffith> todays agenda #link https://wiki.openstack.org/wiki/CinderMeetings
16:00:55 <jgriffith> Shall we?
16:01:00 <jgriffith> jungleboyj: you ready?
16:01:09 <jgriffith> #topic 3'rd party CI
16:01:12 <jungleboyj> jgriffith: Sure.
16:01:32 <jgriffith> jungleboyj: ok, you're on
16:01:45 <jungleboyj> So, just a couple of things I wanted to chat about.  It looks like we didn't land on a decision regarding which tempest tests need to be run.
16:01:52 <jungleboyj> https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification
16:01:56 <avishay> hello
16:02:22 <jungleboyj> The suggesting is to run the "volume" test scenarios and the check-tempest-dsvm-full
16:02:31 <jungleboyj> Is that what we shoueld get running?
16:03:00 <DuncanT> Seems like a good start - until people have tried, we won't know what the issues are
16:03:03 <jgriffith> jungleboyj: cehck-tempest-dsvm-full includes those
16:03:33 <jgriffith> jungleboyj: the question is if people want to just run "volume" tests
16:03:35 <jungleboyj> Ok, so get check-tempest-dsvm-full running.
16:03:41 <jgriffith> jungleboyj: that would be the subset of dsvm-full
16:03:49 <jgriffith> jungleboyj: yes
16:04:10 <DuncanT> We said we'd shoot for dsvm-full every 24 hours or better in the session...
16:04:12 <jungleboyj> jgriffith: Ok.  Thoughts?  DuncanT
16:04:27 * jungleboyj knew you would be the first to chime in.
16:04:29 <jgriffith> DuncanT: that's one thing that's kinda changed a bit after further talks with CI team
16:04:41 <jgriffith> DuncanT: jungleboyj might as well just hit every Cinder commit
16:04:43 <DuncanT> jgriffith: Ah, I missed all of that
16:04:47 <jgriffith> DuncanT: jungleboyj or at least try
16:04:57 <jgriffith> DuncanT: it was an IRC convo after the summit
16:05:14 <jungleboyj> jgriffith: That was what we were leaning towards here.  It is what, 2 or 3 commits an hour on average?
16:05:20 <DuncanT> jgriffith: On #openstack-cinder? I'll go read the log later
16:05:23 <jgriffith> jungleboyj: if that
16:05:40 <jungleboyj> jgriffith: Right,that would be a busy day.
16:05:41 <jgriffith> DuncanT: no, it was either in infra or in the 3'rd party ci weekly meeting
16:05:45 <jgriffith> can't member
16:05:57 <jgriffith> anyway, no reason not to "try" it
16:06:07 <DuncanT> jgriffith: Ok, I'll try to find it, but if we're shooting for every commit, fine by me
16:06:10 <jungleboyj> jgriffith: Partially because of the difficulty of getting useful info out of the daily runs.
16:06:11 <jgriffith> back to your question of dsvm full or other
16:06:22 <jungleboyj> jgriffith: +1
16:06:26 <xyang__> 2 or 3 commits a hour, so one test may haven't finish, another one will start?
16:06:38 <jgriffith> dsvm-full seems like the way to go, however I have to say I'm running into issues with that
16:06:45 <jungleboyj> xyang__: Will be queued up.
16:06:49 <jgriffith> I have EXTREMELY inconsitent results
16:06:53 <jungleboyj> xyang__: potentially.
16:06:54 <bswartz> xyang__: or parallelism!
16:07:04 <jgriffith> random failures in instance creation, networking etc
16:07:12 <bswartz> if you have multiple slaves you can test more than 1 change at a time
16:07:16 <jungleboyj> jgriffith: That isn't good.
16:07:20 <jgriffith> and it's just using the exact infra setup, not using my driver or anything
16:07:33 <jgriffith> jungleboyj: not, it's driving me crazy.  Plus others apparantly have this working
16:07:47 <jgriffith> I keep hoping someone here make some progress on this to compare notes :)
16:08:55 <akerr> Does just the volume tests work for you jgriffith?
16:09:00 <jungleboyj> jgriffith: We haven't gotten that far yet.  We have the right people in the loop and potentially the hardware.  Need the system to put in front of it to be control node/run tests.
16:09:01 <jgriffith> jungleboyj: and NO, not iSCSI only
16:09:16 <jgriffith> akerr: yes, as long as I do a clean inbetween runs
16:09:25 <jungleboyj> jgriffith: Ok, for some reason I thought someone had said that.
16:09:45 <jgriffith> jungleboyj: I think hemna was having issues with that
16:09:49 <jgriffith> jungleboyj: but anyway
16:10:01 <jgriffith> That's the price you pay for doing FC IMO :)
16:10:11 <jungleboyj> jgriffith: :-p
16:10:39 <jgriffith> anybody else actively setting this up right now?
16:11:01 <jungleboyj> jgriffith: So, it sounds like my last question is also answered if we are going to do on each commit.  It will be part of the Jenkins results then.
16:11:12 <jgriffith> jungleboyj: correct
16:11:20 <akerr> jgriffith: I have a toe in the water right now, plan to pick up steam in a week or 2
16:11:22 <jungleboyj> jgriffith: Ok.
16:11:31 <jgriffith> akerr: cool, keep me posted on how it goes
16:11:48 <jgriffith> akerr: are you using the tools laid out by jay?
16:11:55 <akerr> Yes
16:12:09 <jgriffith> akerr: cool, it'll be good to see if maybe it's my crappy AMD boxes :)
16:12:24 * jgriffith has crappy blades with AMD procs
16:12:59 <jgriffith> looks like it's just /me and akerr then for now?
16:13:17 <jgriffith> anything else on this one jungleboyj ?
16:13:24 <jgriffith> did you get answers ?
16:13:51 <jungleboyj> Well, so continue to move forward trying to get dsvm-full working?
16:14:00 <jungleboyj> Or start smaller with the volume tests?
16:14:06 <jgriffith> jungleboyj: dsvm-full
16:14:24 <jgriffith> no reason not to try that first at least
16:14:25 <jgriffith> IMO
16:14:27 <jungleboyj> aye aye captain.  :-)
16:14:43 <kmartin> jgriffith: asselin_ is working on this for us, he'd not in at the moment, but he should be available in the cinder channel in a little bit
16:14:58 <jgriffith> kmartin: cool, I'll be curious to hear how it's going
16:15:14 <jgriffith> Funny, we have 20+ drivers and nobody really working on this yet :(
16:15:17 <DuncanT> Somebody here should be working on it in the next week or so, hopefully not me
16:15:18 <jgriffith> maybe 3 of us
16:15:24 <jgriffith> DuncanT: haha!
16:15:36 <kmartin> jgriffith: I know he is looking at the pci pass though for FC
16:15:37 <jungleboyj> kmartin: Looks like he has been keeping the etherpad updated which is appreciated.
16:15:41 <xyang__> we'll start soon
16:16:03 <jgriffith> coolio
16:16:04 <kmartin> jungleboyj: yes he is updating the etherpad
16:16:12 <jungleboyj> jgriffith: I am hoping that we have people actually trying to set this up in the next couple of weeks.
16:16:23 * jgriffith wishes he had *people*
16:16:28 <jungleboyj> Just pooling resources across the teams now.
16:16:32 <jgriffith> on second thought... no he doesn't
16:16:40 <jungleboyj> jgriffith: :-)
16:16:56 <jungleboyj> jgriffith: Ok, I think that answers my questions.
16:17:02 <jgriffith> jungleboyj: cool
16:17:07 <jgriffith> #topic SSH host keys
16:17:13 <jgriffith> jungleboyj: your's again
16:17:26 <hemna> mornin
16:17:31 <avishay> hemna: yo
16:17:34 <jungleboyj> jgriffith: Yeah ... so, security people here have sunk their teeth into this one.
16:17:42 <jungleboyj> hemna: Buenas Dias
16:18:15 <jgriffith> jungleboyj: so I'm not seeing this as warranting a backport myself?
16:18:16 <jungleboyj> I have gotten them to take a deep breath but want to talk to everyone here about the issue and how we can move forward to a solution.
16:18:24 <jgriffith> jungleboyj: care to ellaborate as to why it should be?
16:18:42 <hemna> jungleboyj, is this one related to the recent ssh patch ?
16:18:54 <hemna> url?
16:18:57 <jgriffith> hemna: see the agenda, there are links
16:19:00 <hemna> k
16:19:05 <jgriffith> https://wiki.openstack.org/wiki/CinderMeetings
16:19:11 <jungleboyj> jgriffith: Ok, so, at this point, I agree on that aspect.  It doesn't do any good without the drivers making fixes.
16:19:22 <jungleboyj> #link https://launchpad.net/bugs/1320050 and #link https://bugs.launchpad.net/cinder/+bug/1320056
16:19:24 <uvirtbot> Launchpad bug 1320050 in cinder "Brocade FC SAN lookup service should allow customized hosts key and missing policy" [High,Fix committed]
16:19:25 <hemna> thnx
16:19:39 <jgriffith> What does the driver need to fix?
16:19:50 <jungleboyj> Let me take a step back.
16:19:56 <DuncanT> Looks like a legitimate bug, my main concern is, as usual, keeping running configs running
16:20:00 <jgriffith> I think it's kinda cumbersome to not share keys independent of driver
16:20:14 <jgriffith> DuncanT: which one are you looking at :)
16:20:44 <DuncanT> jgriffith: Both actually ;-)
16:20:50 <jgriffith> DuncanT: :)
16:21:02 <edmondsw> by not checking the known_hosts file and using RejectPolicy when it doesn't match, you leave yourself open to MITM (man-in-the-middle) attacks
16:21:06 <jungleboyj> edmondsw: Cna you chime in.
16:21:12 <edmondsw> we need to fix that throughout
16:21:28 <jungleboyj> jgriffith: DuncanT  ^^^ The person who found this.
16:21:28 <DuncanT> edmondsw: Low risk in most environments, but agreed
16:21:33 <jgriffith> edmondsw: I guess I didn't read the bug that way
16:21:51 <jgriffith> edmondsw: DuncanT to me this sounded like "I want to have my custom keys"
16:22:08 <jgriffith> edmondsw: ie not have them in the hosts ~/.ssh dir?  No?
16:22:09 <DuncanT> edmondsw: The problem is that if we change the default the currently working installs grind to a halt, don't they? Or will the auto-add have stashed the key by then?
16:22:25 <jungleboyj> jgriffith: The new thing that came out of this is wanted to have a config option to support this.
16:22:35 <edmondsw> jgriffith: might as well fix that while we're at it, but not, the issue is bigger than using a custom known_hosts file
16:22:37 <markstur__> auto-add should have stashed the keys by then.
16:22:53 <DuncanT> markstur__: Thanks
16:22:57 <markstur__> but what they "used to do" won't work the next time in a clean env
16:23:35 <DuncanT> markstur__: I'm far less concerned about that, though we could still default to auto-add and let packagers harden if they want
16:23:47 <edmondsw> right now some things are using auto-add, some things are using warnings (which log but then continue). I'm not sure anything is actually rejecting when keys don't match, that I saw
16:23:49 <asselin_> hi
16:24:12 <jungleboyj> asselin_: Welcome.
16:24:14 <jgriffith> edmondsw: indeed... ok that I agree is poor
16:24:27 <jgriffith> I'd propose we focus on fixing the lack of security here first
16:24:42 <jungleboyj> jgriffith: +1
16:24:44 <jgriffith> then revisit the idea of letting each driver add a custom key file/location
16:25:18 <markstur__> missing key policy should be configurable somewhere not hard-coded
16:25:47 <jgriffith> markstur__: I think what I'm saying is there shouldn't be a "missing key policy" per say
16:25:56 <jgriffith> you don't have the keys, or the keys don't match and your fail
16:26:02 <jgriffith> s/your/you/
16:26:23 <avishay> i.e., policy == reject
16:26:25 <edmondsw> jgriffith: my thought was that by supporting config settings to indicate what policy (and known_hosts... why not add that together?) you don't necessarily break anyone.
16:26:29 <jungleboyj> jgriffith: That sounds like a safer default right now.
16:26:39 <edmondsw> Could leave defaults as they are today, initially, and then revisit the defaults when we can work out how to not break folks
16:26:43 <jgriffith> avishay: yeah
16:26:54 <jgriffith> then why bother?
16:27:14 <markstur__> jgriffith:  So RejectPolicy is the missing_key_policy -- but really it should be configurable for at-your-own-risk type environments (test)
16:27:23 <jgriffith> honestly unless I'm missing something why have an option that says "you can do this insecurely if you want"
16:27:36 <jgriffith> markstur__: why?
16:27:41 <edmondsw> jgriffith: this allows operators to harden, without breaking anyone. Doesn't mean it's a final solution, but buys time to work out how to fully fix
16:27:48 <jgriffith> markstur__: we don't do that for instances for example
16:28:14 <jgriffith> edmondsw: I guess I don't understand why we need to "buy time" rather than just fix it
16:28:17 <edmondsw> jgriffith: we have other options to allow you to do things insecurely :)
16:28:29 <jgriffith> it seems everybody feels this is poor / lacking in the realm of security features
16:28:52 <bswartz> +1 for having an option to do it insecurely for test/dev environments
16:28:57 <jgriffith> edmondsw: ok... well, I guess I should defer to folks that actually need/use this :)
16:29:02 <hemna> bswartz, +1
16:29:06 * bswartz sets his keystone password to "password" in his dev environment
16:29:06 <jgriffith> bswartz: can anybody explain why?
16:29:11 <jgriffith> bswartz: hemna ?
16:29:11 <jungleboyj> edmondsw: I guess I am a little confused.  Initially you were worried about security.  So, why not just harden security?
16:29:14 <DuncanT> jgriffith: Breaking every install doc ever would be a downside of changing it to reject-by-default
16:29:31 <jungleboyj> DuncanT: Details.  ;-)
16:29:42 <jgriffith> DuncanT: wouldn't be the first time :)  frankly I'm not too worried about that, you just update docs
16:29:59 <jgriffith> If we never "fixed" anything because we didn't want to update docs that would be sad
16:30:00 <edmondsw> jungleboyj: yeah, I don't mean to come off the wrong way here... I was trying to stage the solution to help out
16:30:04 <jungleboyj> jgriffith: Plus the failure should be pretty clear.
16:30:14 <hemna> engineers update docs?
16:30:17 <hemna> :P
16:30:25 <jgriffith> hemna: bswartz still wondering about the advantage for a test option here?
16:30:31 <avishay> hemna: just put docimpact and it's magic
16:30:36 <jungleboyj> edmondsw: The complication with new config options is that it really turns into a new feature and would be Juno only.
16:30:41 <jgriffith> avishay: +1 :)
16:30:43 <jungleboyj> No way to backport.
16:30:47 <hemna> avishay, :)
16:30:53 <jgriffith> jungleboyj: yes and that's how it should be IMO
16:30:54 <DuncanT> jgriffith: You asked for an example of why you might not want to go default=reject straight away. I'm fine with new installs breaking as long as upgrades don't, and it sounds like the key /should/ be cached for all running installs, so we're good there
16:31:04 * jungleboyj likes magic
16:31:07 <edmondsw> jungleboyj: can't be Juno only... we need to backport
16:31:11 <jgriffith> DuncanT: uh oh... are we agreeing again :)
16:31:16 <jgriffith> edmondsw: why?
16:31:17 <bswartz> if forcing a secure option is not a large burden I'm not opposed to it but I know that people tend to take the path of least resistance when testing/developing and I don't see a problem with that
16:31:23 <jgriffith> edmondsw: It doesn't fit the criteria IMO
16:31:27 <jgriffith> edmondsw: it's a feature
16:31:29 <markstur__> jgriffith:  re: why? -- Seems to me that if we harden someone, somewhere will need to loosen or write their own known host auto-add mechanisms.  That is usually the intent, but it seems like a config is the better work-around.  Does that answer the "why" way back up there that I missed?
16:31:45 <jgriffith> Ok... let's back up
16:31:47 <edmondsw> jgriffith: security vulnerability, not feature
16:31:49 <DuncanT> jgriffith: The advantage of being able to config it off is that for frequently reinstalled systems (e.g. gate), having to go fetch the key, outside of cinder, is a pain in the ass
16:32:02 <jgriffith> I had two people that said "needs to be configurable to make their lives easier"
16:32:05 <jgriffith> hemna: bswartz
16:32:11 <jgriffith> why does this matter for you?
16:32:24 <jgriffith> as opposed to generating a key in the driver and actually using it?
16:32:47 <hemna> for testing/dev environments.   I suppose we could automate it via a devstack local.sh though
16:32:51 <edmondsw> jgriffith: "configurable to make their lives easier" can be a feature... but not rejecting when known_hosts doesn't match is a vulnerability
16:32:52 <DuncanT> generating a key in the driver? How do you confirm that's the key on the device? That's the whole point....
16:32:53 <bswartz> I'm just throwing an opinion out there -- it doesn't matter to me
16:33:10 <jgriffith> DuncanT: maybe I'm missing something here....
16:33:23 <jgriffith> DuncanT: I had assumed we'd have the driver setup keys during init
16:33:24 <bswartz> I don't use SSH so this won't impact me one way or the other
16:33:30 <jgriffith> bswartz: haha
16:33:35 <hemna> hehe
16:33:40 <DuncanT> jgriffith: No, that doesn't solve the MITM problem
16:33:53 <jungleboyj> DuncanT: +1
16:33:58 <jgriffith> DuncanT: ok, I'm missing something here
16:34:14 <DuncanT> jgriffith: To do it securely, you have to set up the keys outside of cinder, having confirmed they are really the ones from the SAN
16:34:18 <edmondsw> jgriffith: "have the driver setup keys during init" meaning have it auto-add to known_hosts the first time it talks to the device?
16:34:25 <jgriffith> DuncanT: ok, so that's fine as well IMO
16:34:34 <jgriffith> DuncanT: have devstack setup do it
16:34:45 <jgriffith> edmondsw: yes, that's what I was thinking
16:34:53 <jgriffith> edmondsw: and I'm still unclear on why that doesn't work
16:35:06 <jgriffith> but I'm not arguing :)
16:35:12 <DuncanT> jgriffith: auto-add by default means you can get hit by MITM at install time
16:35:14 <avishay> jgriffith: someone can MITM during setup
16:35:17 <edmondsw> jgriffith: better, but could still have had a MITM attack in progress when you make that first connection
16:35:25 <jgriffith> avishay: how would they do that?
16:35:38 <jgriffith> edmondsw: ahh... got it
16:35:39 <jungleboyj> avishay: True.
16:35:40 <jgriffith> ok
16:35:53 <jgriffith> seems like a bit of a strecth to me but that's fine
16:35:54 <DuncanT> jgriffith: ARP spoof to steal the SAN IP, and fake a SAN?
16:36:01 <avishay> jgriffith: we're talking 1337 h4x0rs here :)
16:36:12 <jgriffith> I'd still just say we make strict key checking by default and set it up in devstack for devs
16:36:13 <jungleboyj> Granted all of this is somewhat unlikely in provate clouds but once we have more hybrid clouds it becomes a bigger concern.
16:36:26 <jgriffith> jungleboyj: hybrid clouds... surely you jest!
16:36:29 <jgriffith> :)
16:36:35 <DuncanT> jgriffith: auto-add on first connect as a default, with a fail-if-changed policy, matches your idea
16:36:41 <jungleboyj> jgriffith: Someone said they are coming.  ;-)
16:36:49 <edmondsw> this is why I was assuming we'd need a conf setting to specify the policy and the pre-configured known_hosts file to check against... so operator can set that up prior to first connect
16:36:50 <jgriffith> DuncanT: correct
16:36:59 <DuncanT> jgriffith: And have a config option for super-strict you-sort-all-keys-in-advance mode
16:37:04 <jgriffith> DuncanT: but honestly I'm also fine with stricter impl as well
16:37:11 <jgriffith> and I'm failing to see why it's sooooo hard?
16:37:11 <DuncanT> jgriffith: For paranoid installs
16:37:18 <bswartz> time check -- this topic has taken 20 minutes
16:37:22 <jgriffith> just configure devstack to generate and setup keys
16:37:26 <jgriffith> update docs
16:37:26 <jgriffith> done
16:37:30 <hemna> edmondsw, so an admin is going to have to build the known_hosts file for every SAN prior to starting Cinder ?
16:37:30 <avishay> i think there is consensus
16:37:35 <DuncanT> devstack is not the only fruit
16:37:43 <jgriffith> bswartz: you're just worried navneet won't get enough time :)
16:37:52 <bswartz> jgriffith: yes!
16:37:56 <jgriffith> bswartz: :)
16:37:57 <navneet> :)
16:38:08 <jgriffith> bswartz: navneet we'll cut over at 20 til
16:38:09 <avishay> he is also worried that jgriffith won't have time
16:38:10 <glenng> *chuckles*
16:38:14 <edmondsw> hemna: if they want the strict security, yes. Could be configurable to allow less strictness, e.g. auto-add on first connect
16:38:17 <jungleboyj> jgriffith: So, can we try to put together a quick plan for this?
16:38:18 <jgriffith> hemna: do you want security or not :)
16:38:28 <hemna> :P
16:38:40 <jungleboyj> jgriffith: It sounds like this is something that we need to create a Blueprint for.  Would you agree?
16:38:46 <jgriffith> edmondsw: that's a compromise at any rate.  I could live with that but I don't see the point at all
16:38:59 <jgriffith> especially if you all are saying "we have to backport"
16:39:05 <DuncanT> jgriffith: This is not a binary decision - turning security decusions into those tends to make from bad conclusions
16:39:10 <jgriffith> then it seems it's not really that big of an issue IMO
16:39:18 <hemna> so can we at least dump something useful in the logs then, so an admin's job of building the known_hosts file isn't a mystery
16:39:20 <edmondsw> my concern is strict security... if you want to have a config setting to allow less strictness, that's fine, but I won't use it :)
16:39:22 <jgriffith> DuncanT: fair enough
16:39:49 <jgriffith> DuncanT: although I'd say if someobdy files an OSS and says "OMG security issue, must backport to everywhere" then it does become a bit binary
16:39:54 <jungleboyj> DuncanT: +2 ... I want to make sure we talk through this and com eup with a good solution.
16:39:56 <jgriffith> Just sayin :)
16:40:16 <jgriffith> Honestly I don't use SSH so not sure why I care ;)
16:40:21 <DuncanT> jgriffith: Security bods tend to suggest the sky is falling for whatever issue they've just found
16:40:34 <avishay> time
16:40:36 <hemna> edmondsw, +1.   If it's configurable, then that makes test/dev environments easy to deal with.   just set the auto-add config setting and away you go.
16:40:38 <jgriffith> Oh.... wait... I remember,  because Cinder shouldn't be a steaming pile o'crap
16:41:07 <jgriffith> hemna: just adding 3 lines of code to devstack/lib/cinder makes it even easier!
16:41:09 * jungleboyj notes that I would like to get a plan to go forward.
16:41:41 <jgriffith> Ok....
16:41:43 <jgriffith> time's up
16:41:44 * DuncanT suggests default to auto-add on first connect, have a config option for super-paranoid mode
16:41:46 <jgriffith> but real quick
16:41:51 <jgriffith> DuncanT: +1
16:41:59 <jgriffith> jungleboyj: edmondsw does that sound ok to you?
16:42:15 <jgriffith> default auto-add on first connect, option to harden?
16:42:22 <markstur__> I think security cops would not like the auto-add default
16:42:23 <edmondsw> DuncanT: +1 if that config option is two... policy and known_hosts location
16:42:29 <hemna> DuncanT, +1
16:42:41 <jgriffith> markstur__: I agree, but you guys keep shooting me down because apparantly it's "toooo hard"
16:42:47 <hemna> markstur__, if it's configurable, then the security cops can be happy.
16:42:49 <jungleboyj> DuncanT: jgriffith So, we keep current config for first connect and throw an error for subsequent changes.
16:42:50 <DuncanT> edmondsw: Fine with me
16:42:53 * jgriffith uses his 'whinie' voice
16:43:17 <DuncanT> jungleboyj: +2
16:43:18 <edmondsw> and if that auto-add is only on first connect, and it will reject thereafter by checking against known_hosts
16:43:22 <jgriffith> Ok... jungleboyj edmondsw markstur__ we good at least for a start?
16:43:46 <jungleboyj> jgriffith: Sounds good to me.  Should we do this through a cinder-spec then?
16:43:47 <jgriffith> navneet: I'm trying buddy :)
16:43:55 <jgriffith> jungleboyj: yes, it's def a feature
16:43:58 <navneet> jgriffith: i understand :)
16:44:04 <jungleboyj> jgriffith: No backport.
16:44:17 <jgriffith> jungleboyj: it's a feature, so that would mean no
16:44:23 <edmondsw> jgriffith: need backport... have to fix security vulnerability in released versions
16:44:25 <jgriffith> jungleboyj: but I'll let others make a fuss if they like
16:44:30 <jgriffith> haha
16:44:38 <jgriffith> edmondsw: jungleboyj let's chat in #cinder after meeting
16:44:38 <edmondsw> I can make a fuss :)
16:44:41 <jungleboyj> jgriffith: Ok, warned edmondsw that was going to be the answer.
16:44:45 <avishay> OK, how about first come up with a solution, then see about backport?
16:44:51 <jungleboyj> jgriffith: Ok.
16:44:53 <jgriffith> avishay: fair
16:45:05 * avishay looks at his watch :)
16:45:06 * jungleboyj yields the floor.
16:45:09 <jgriffith> #topic Dynamic multi-pools
16:45:13 <jgriffith> navneet: you're up
16:45:22 <navneet> jgiffith: the code is WIP
16:45:30 <navneet> if somebody wants to gv early feedback
16:45:34 <navneet> https://review.openstack.org/#/c/85760/
16:45:43 <navneet> its nearly done
16:45:52 <avishay> navneet: what does WIP mean?  still writing?  can test?
16:46:02 <bswartz> the main change since atlanta is added unit tests
16:46:08 <navneet> avishay: I was but m done now for c-vol
16:46:20 <navneet> some differences still not submitted
16:46:24 <navneet> ll do it soon
16:46:28 <avishay> OK
16:46:47 <hemna> I'd still prefer a simpler approach that simply uses pools in volume types and pass the pool stats to the scheduler in the get_volume_stats
16:46:59 <DuncanT> hemna: +1
16:47:02 <jgriffith> hemna: +1
16:47:18 <navneet> hemna: that does no sound like a clean approach
16:47:22 <jgriffith> Mabye one of use should propose an alternate and we can all compare/contrast?
16:47:28 <navneet> you need to make changes at many places
16:47:37 <hemna> I think if we do the simpler approach and it then still doesn't meet the needs of folks, we can try navneet's approach or something similar.
16:47:42 <navneet> also the rpc layer
16:47:42 <jgriffith> as long as the end result is the same should be ok no?
16:47:46 <bswartz> hemna: that may sound simpler but the code change would be a lot more and scattered around
16:48:00 <jgriffith> bswartz: but that's the OS way!!! :)
16:48:04 <DuncanT> navneet: Your approach changes the way the driver is called, in quite fundamental ways, I'm still far from convinced there aren't serious issues
16:48:08 <navneet> bswartz: +1
16:48:19 <kmartin> navneet: we have been using that approach for sometime
16:48:20 <jgriffith> bswartz: seriously though, it might open the door to some other 'features/enhancements' in the future
16:48:24 <jgriffith> it's more flexible
16:48:32 <navneet> DuncanT: u can highlight the issues
16:48:34 <DuncanT> navneet: Changing the driver from a singleton to having many is a big deal IMO
16:48:45 <jgriffith> DuncanT: +1
16:48:45 <hemna> DuncanT, +1
16:48:54 <navneet> DuncanT: we do ti for multiple backends today
16:48:56 <guitarzan> doesn't multibackend already do that?
16:49:01 <navneet> we r just extending it for pools
16:49:10 <jgriffith> navneet: actually no I don't think so
16:49:12 <avishay> driver is not aware of multi-backend
16:49:15 <DuncanT> navneet: Yeah, but they're talking to different backends
16:49:23 <jgriffith> navneet: guitarzan driver is still a singleton
16:49:29 <bswartz> navneet, DuncanT: the difference is multiple instances of the driver in one PID vs in different PIDs
16:49:45 <bswartz> you can make multiple instances of a driver talk to a single backend today using multiple PIDs
16:49:48 <DuncanT> navneet: e.g. now there'll be (NPOOLS x ssh connections) rather than one
16:49:48 <navneet> DuncanT: you can have same endor and controller as diff backends too
16:49:59 <jgriffith> sighhh...
16:50:38 <DuncanT> bswartz: You can do that, but if you break it you get to keep both halves... you can't do it by accident
16:50:38 <navneet> DuncanT: NPOOLs x ssh for management?
16:50:41 <jgriffith> Ok... so here's the thing
16:50:51 <jgriffith> bottom line is that we need the capability and we all agree on that
16:50:57 <hemna> yup
16:50:57 <jgriffith> implementation is still questionable
16:51:02 <DuncanT> jgriffith: +1
16:51:03 <bswartz> does someone want to volunteer to write the feature in the way hemna suggests?
16:51:06 <jgriffith> I'd propose we all take a look at the WIP
16:51:18 <jgriffith> test it, inspect it, try to break it
16:51:25 <navneet> jgriffith: dats wat I wnt
16:51:29 <hemna> bswartz, If I had the time I'd do it, but I'm buried right now.
16:51:32 <avishay> jgriffith: +1 - no point in debating before looking at the code
16:51:33 <bswartz> we could do it, but it will take time and we've already spent a bunch of time getting it right using the way we suggested
16:51:39 <jgriffith> and if possible maybe look at an alternative approach via a prototype
16:51:56 <jgriffith> bswartz: I wouldn't expect you to have to do it
16:52:05 <DuncanT> Gonna need time to beat on it... can we agree not to merge it before next meeting? I'd be happy with that as a starting point
16:52:09 <jgriffith> bswartz: you already proposed a solution as far as I'm concerned
16:52:18 <jgriffith> DuncanT: +1000
16:52:27 <navneet> jgriffith: dats fine
16:52:31 <jgriffith> I have no intention of approving that patch in the next week
16:52:41 <DuncanT> Cool, I'll relax then
16:52:54 <jgriffith> It's still a WIP so that shouldn't be a real risk
16:52:58 <navneet> jgriffith: I want to talk abt cinder back up
16:53:03 <avishay> so let's all try to do a high-level review at least, think if there is a better alternative, and reconvene next week?
16:53:08 <jgriffith> navneet: you have two minutes
16:53:14 <jgriffith> avishay: that sounds good
16:53:14 <DuncanT> avishay: +1
16:53:19 <navneet> the backup is not written right
16:53:21 <jgriffith> #topic cinder-backup
16:53:26 * bswartz notices jgriffith saving time for his own topic
16:53:30 <jgriffith> navneet: haha... you know how to make friends
16:53:31 <navneet> as far as back up manager is concerned
16:53:46 <DuncanT> navneet: Details?
16:53:53 <navneet> jgriffith: it does a lot of routing stuff in the manager
16:53:57 <avishay> DuncanT: stop being so defensive ;)
16:54:05 <navneet> which should be handled at the rpc layer
16:54:23 <navneet> DuncanT: just said
16:54:26 <DuncanT> navneet: scheduler IMO
16:54:34 <navneet> it routes calls to backends in the mgr
16:54:45 <navneet> no manager
16:54:51 <bswartz> navneet is referring to the way the c-bak service handles incoming RPC messages
16:54:52 <avishay> navneet: yes, because there is no backup scheduler
16:55:00 <bswartz> the code is in the wrong place
16:55:15 <navneet> avishay:
16:55:16 <bswartz> and he's proposing a fix
16:55:20 <navneet> yes
16:55:34 <navneet> the rpc calls all got the same host
16:55:37 <DuncanT> navneet: It was done that way because it broke otherwise - originally there was a backup scheduler, but it broke stuff... most of the stuff is now fixed, I'd be totally fine with bringing back a backup scheduler
16:55:39 <navneet> which needs to change
16:55:46 <navneet> and go as per the backends
16:55:54 <jgriffith> navneet: cool
16:56:06 <jgriffith> navneet: sounds like someone interested should propose a spec/bp
16:56:09 <bswartz> anyways there's not much point talking about it before the code is available, so expect that soon as well
16:56:13 <navneet> jgriffith,DuncanT: m proposing a fix
16:56:17 <jgriffith> bswartz: actually
16:56:33 <navneet> tchange it to handle msg better
16:56:34 <jgriffith> navneet: great...  spec/bp please
16:56:41 <navneet> jgriffith: sure
16:56:44 <avishay> 4 min
16:56:54 <jgriffith> anything else navneet ?
16:56:55 <DuncanT> navneet: Certainly I'm interested in seeing it... I agree on the issue existing
16:57:07 <navneet> jgriffith: applies to pools support for back up too
16:57:18 <jgriffith> navneet: I got that :)
16:57:29 <navneet> jgriffith: ok ll propose a blueprint
16:57:33 <jgriffith> navneet: but this is an example where I think the "other" method would be cleaner
16:57:45 <avishay> speaking of proposing blueprints...
16:57:50 <navneet> jriffith: no thats not true
16:57:50 <jgriffith> avishay: LOL
16:57:57 <navneet> current ways are a mess
16:57:58 <bswartz> the fix should be small
16:57:58 <jgriffith> navneet: ok... ok
16:58:03 <avishay> jgriffith: how do we propose specs?
16:58:09 <avishay> jgriffith: what is the process?
16:58:12 <jgriffith> #topic specs and bp's
16:58:16 * jungleboyj has one in process now
16:58:18 <jgriffith> avishay: I'm glad you asked :)
16:58:24 <avishay> jgriffith: :)
16:58:32 <jgriffith> So the idea is something like this
16:58:50 <jgriffith> You may have noticed that BP's sometimes go into a bit of a black hole
16:59:01 <jgriffith> I miss things (what can I say, I'm human)
16:59:12 <jgriffith> and LP sucks for trying to search/sort
16:59:22 <jgriffith> and LP doesn't force things like good details when proposing a bp
16:59:25 <jgriffith> Soooooo
16:59:29 * bswartz has noticed the blackhole phenomenon on LP
16:59:29 <jgriffith> cinder-specs
16:59:42 <jgriffith> https://review.openstack.org/#/q/status:open+cinder-specs,n,z
16:59:52 <jgriffith> The idea is that it does not replace BP's
16:59:54 <jgriffith> BUT
16:59:54 <DuncanT> We using the nova template initially?
17:00:02 <jgriffith> it triggers targetting of BP's
17:00:16 <jgriffith> and get's the restof the Cinder team actively engaged in reviewing proposals
17:00:26 <jgriffith> best of all it forces some additional clarity and detail
17:00:33 <jgriffith> The process goes like this:
17:00:39 <jgriffith> 1.  Submit a BP
17:00:44 <jgriffith> 2. Submit a detailed spec
17:00:50 <bswartz> is it smart to put the specs in a whole different repo?
17:00:54 <jgriffith> ie: git clone openstack/cinder-specs
17:01:01 <jgriffith> git chekcout -b my-change
17:01:12 <jgriffith> add your yaml file blah blah blah
17:01:17 <jgriffith> git commit -a
17:01:20 <jgriffith> git review
17:01:33 <jgriffith> 3. Cinder folks get to review, comment etc etc
17:01:44 <jungleboyj> jgriffith: So we need to submit the BP before creating the Spec?
17:01:47 <jgriffith> If / When approved spec gets approved and merges
17:01:51 <jgriffith> jungleboyj: yes
17:01:56 <jungleboyj> jgriffith: Ok.
17:02:01 <jungleboyj> Good to know.
17:02:01 <jgriffith> then jgriffith goes in an targets the BP
17:02:19 <jgriffith> anything that's not targetted will now just "fall" off the radar on the milestone targets
17:02:26 <jgriffith> we'll cull them automagically
17:02:55 <jgriffith> and avoid this business of folks adding/approving their own BP's willy nilly without me or anybody else konwing about them
17:02:58 <jgriffith> make sense?
17:03:01 <avishay> reviewers - make sure you add the new repo to your gerrit filters so you don't forget about those patches (like what happens with cinderclient) :)
17:03:01 <garyk> is there the vmware meeting now?
17:03:03 <bswartz> +1
17:03:09 <avishay> garyk: finishing up
17:03:14 <jgriffith> garyk: yeah.. sorry
17:03:15 <garyk> avishay: thanks!
17:03:17 <jgriffith> I'm late :(
17:03:20 <garyk> np. take your time
17:03:22 <jgriffith> ok... that's all I've got
17:03:33 <avishay> jgriffith: awesome, like the idea
17:03:36 <jgriffith> you can hit me up in #openstack-cinder for question
17:03:43 <jgriffith> avishay: I think it will help A LOT
17:03:47 <avishay> agreed
17:03:54 * jgriffith wishes he could take credit for the idea :(
17:03:56 <jgriffith> Ok...
17:03:57 <avishay> :)
17:03:59 <jgriffith> thanks everybody!!!!
17:04:03 <jgriffith> #endmeeting