16:00:07 <thingee> #startmeeting cinder
16:00:08 <openstack> Meeting started Wed Dec 10 16:00:07 2014 UTC and is due to finish in 60 minutes.  The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:12 <openstack> The meeting name has been set to 'cinder'
16:00:15 <thingee> hello all
16:00:20 <rushiagr> hi!
16:00:27 <tbarron> hi
16:00:30 <patrickeast> hi
16:00:31 <kfox1111> hi
16:00:37 <eikke> hello
16:00:44 <thingee> today's agenda is pretty full
16:00:45 <mtanino> hi
16:00:46 <thingee> #link https://wiki.openstack.org/wiki/CinderMeetings
16:00:48 <bswartz> hi
16:00:49 <e0ne> hi
16:01:06 <dulek> hi
16:01:15 <thingee> Quick note to reviewers for Cinder, drivers get priority this milestone
16:01:15 <dustins> \o
16:01:19 <thingee> #link https://etherpad.openstack.org/p/cinder-kilo-priorities
16:01:22 <smcginnis> o/
16:01:27 <cknight> Hi
16:01:32 <thingee> it's important for core to help out and help see a driver through or two
16:01:39 <scottda> hi
16:01:40 <rhe00_> hi
16:01:49 <thingee> I would like to see these merged this week
16:01:59 <thingee> even if you're not core, please help
16:02:02 <Swanson> hello
16:02:06 <jgriffith> thingee: What's your take on Tempest tests?
16:02:11 <ameade_> o/
16:02:27 <jgriffith> Personally I'm not inclined to review if they don't pass the full Volume tests and show it
16:02:27 <thingee> jgriffith: they're great
16:02:33 <thingee> jgriffith: ah heh
16:02:33 <jgriffith> thingee: ok
16:02:36 <jgriffith> thingee: thanks
16:02:49 <thingee> jgriffith: sorry not trying to be smart. I wasn't sure what you were talking about
16:02:56 <jgriffith> thingee: :)
16:02:56 <eikke> is it OK to add a driver to the K-1 drivers list?
16:03:06 <jordanP> https://etherpad.openstack.org/p/cinder-kilo-priorities ^^
16:03:24 <thingee> jgriffith: they should definitely pass the full volume tests in tempest. that was a requirement as mentioned in the email
16:03:29 <avishay> o/
16:03:36 <flip214> jgriffith: so you'd prefer that drivers faked things because dependencies don't provide all required functions yet?
16:03:38 <thingee> anyone can edit the etherpad if a driver actually isn't ready to be reviewed
16:03:41 <Swanson> Unit tests for the Dell driver are going in today.
16:03:54 <thingee> Swanson: thanks
16:04:04 <thingee> ok lets start with the first item
16:04:04 <jordanP> thingee, what if the driver is ready to be reviewed ?
16:04:10 <jgriffith> flip214: No
16:04:13 <jordanP> thingee, but not on the list
16:04:19 <thingee> jordanP: if it's not on the list, add it
16:04:24 <jordanP> right
16:04:35 <thingee> #topic Abstract interface for volume drivers
16:04:39 <thingee> mkoderer: here?
16:05:03 <thingee> #link https://review.openstack.org/#/c/127455/
16:05:11 <thingee> so that's the review that's sitting ^
16:05:36 <thingee> I want to know what the rest of the folks think of drivers having to adopt this
16:05:39 <thingee> #link https://review.openstack.org/#/c/138661/2/cinder/volume/drivers/rbd.py
16:05:41 <flip214> +1
16:05:45 <thingee> that's an example of the rbd driver with it
16:05:52 <thingee> discuss
16:05:56 <jungleboyj> o/
16:06:20 <hemna_> morning
16:06:22 <eikke> thingee: what would the timeframe be?
16:06:26 <patrickeast> i was curious what the impact will be for drivers that inherit from the iscsi/san driver base class
16:06:36 <thingee> eikke: to adopt it? none yet.
16:06:38 <flip214> I think that's a nice feature-detecting implementation
16:06:51 <asselin> o/
16:06:55 <thingee> patrickeast: explain
16:06:59 <eikke> flip214: +1
16:07:02 <DuncanT> Timeline would be best-effort for Kilo, I think
16:07:10 <e0ne> should all new drivers in K use it?
16:07:12 <patrickeast> well the rbd example shows one that uses just the volume driver directly
16:07:45 <thingee> patrickeast: I think it would be the same?
16:07:57 <thingee> it's the core features that alll drivers need to have today
16:08:04 <akerr> patrickeast: your driver should inherit the features from the parent, and then you would need to add any other extra features you need
16:08:18 <patrickeast> ok got it
16:08:43 <thingee> so IMO, adopting this shouldn't be difficult. I think just like third party CI we need good communication and reasonable time
16:09:04 <avishay> seems like an easy enough change
16:09:28 <thingee> if we agree with this change, we can communicate to maintainers by the end of kilo
16:09:37 <thingee> that's two milestones...
16:09:41 <thingee> anyone disagree?
16:09:46 <jgriffith> Just FYI I gave it a -1 because it introduces merge conflicts for items in progress
16:09:47 <smcginnis> +1
16:09:52 <ameade_> +1 sounds easy enough
16:09:57 <hemna_> thingee, +1
16:10:21 <thingee> does anyone disagree with this change and approach?
16:10:23 <rushiagr> +1 (agree)
16:10:24 <tbarron> +1
16:10:30 <eharney> no
16:10:38 <DuncanT> jgriffith: merge conflicts happen - if something beats you to the merge then you have to fix up IMO - adding -1 seems a bit unfair
16:10:50 <jgriffith> DuncanT: not unfair to me
16:10:53 * jungleboyj came in late.  We have to go back and look.
16:10:53 <eikke> wouldnt it make sense to only merge this after K-1 closed?
16:10:54 <thingee> ok, I will handle communication once this is merged in k-1.
16:10:58 <jgriffith> DuncanT: feel free to ignore my -1
16:11:03 <eharney> DuncanT: i was thinking the same thing, merge conflicts can get resolved at merge time
16:11:19 <thingee> eikke: sure
16:11:41 <thingee> Ok I just wanted to understand if the rest of the team agreed with this approach and should we continue to move forward
16:11:43 <thingee> thanks
16:11:45 <jgriffith> eikke: +1
16:11:59 <thingee> ok so we'll merge after k-1
16:12:02 * jgriffith agrees with the apparoach for sure
16:12:06 <thingee> #topic CI for zone drivers?
16:12:13 <thingee> jnrao: here?
16:12:16 <jnrao> yes
16:12:28 <thingee> DuncanT: I know you had thoughts on this from the summit too
16:12:43 <jnrao> i would like to know, is there any framework in place for us to start CI for zone drivers?
16:13:01 <hemna_> jnrao, what specifically do you mean by zone drivers ?
16:13:19 <DuncanT> thingee: My only thoughts were 'we should totally do this'
16:13:23 <hemna_> the fibre channel zone manager drivers (cisco and brocade) ?
16:13:32 <jnrao> Whjyes
16:13:33 <jnrao> yes
16:13:46 <bswartz> fibre channel switches/networks have something called zones for those that don't know
16:14:00 <hemna_> jnrao, the problem now is that the CI system can't do FC
16:14:01 <bswartz> it's a FC-specific security construct
16:14:08 <hemna_> because libvirt doesn't support FC NPIV
16:14:17 <jnrao> ic
16:14:22 <hemna_> it's possible
16:14:25 <hemna_> but it's a major pain
16:14:36 <thingee> #idea should cinder have third-party ci for FC zone drivers?
16:14:41 <hemna_> we have to do a PCI passthrough for our HBA's in our local CI in order to test our FC volume drivers
16:14:46 <asselin> hemna_, we have CI doing FC
16:14:46 <bswartz> the problem is that nothing in openstack manages FC networks, so the job falls on cinder
16:14:53 <hemna_> asselin, yes I know
16:15:26 <hemna_> sorrison, I like the idea of having CI for the FCZM and it's drivers and the lookup services.
16:15:31 <hemna_> *sigh*
16:15:33 <hemna_> so
16:15:40 <hemna_> but it's on brocade and cisco to do it.
16:15:47 <jungleboyj> hemna_: +1
16:16:01 <avishay> hemna_: makes sense to me
16:16:03 <asselin> hemna_, agree +1
16:16:07 <jgriffith> hemna_: +1, although I'd argue it should also fall on Vendors that consume it to a point
16:16:13 <hemna_> sorrison, not only do they need to have tests for the drivers, but they will also need an FC array to do the complete tests.
16:16:21 <DuncanT> I think that all the people with a stake in FC should definitely be looking at CI - not as a requirement yet, but as something that would make things better for sure
16:16:26 <jnrao> I was looking out for core team to provide some guidelines for FC vendors like us do the CI.
16:16:26 <hemna_> jgriffith, yah, we are doing some tests with it,
16:16:27 <bswartz> I agree with jgriffith -- it seems unlikely that cisco and brocade will start caring about cinder
16:16:37 <jgriffith> hemna_: in other words, if you want an FC driver in OpenStack you're part of it
16:16:53 <hemna_> jgriffith, yup.  we do testing with the FCZM and one of the drivers now.
16:17:02 <jgriffith> hemna_: and like anything, you're going to need CI that demonstrates you work on both
16:17:02 <hemna_> but we don't have both cisco and brocade switches
16:17:07 <hemna_> so we are only testing with 1 currently
16:17:10 <DuncanT> jnrao: Do you have the facilities to run CI?
16:17:15 <xyang1> we test FCZM too
16:17:15 <jgriffith> hemna_: cuz that's the beauty of FC, it's not all the same necessarily
16:17:21 <jnrao> we are setting up CI for cinder.
16:17:40 <akerr> we're in planning stages for fc ci
16:17:42 <jnrao> So we need some guidelines on how to do the CI with zone drivers.
16:17:46 <jgriffith> All vendors send a device to jnrao to test ;)
16:17:51 <hemna_> hehe
16:18:06 <hemna_> jnrao, do you have an FC enabled array?
16:18:17 <jnrao> i mean we do only for Brocade switch gear.
16:18:22 <jnrao> not for all.
16:18:33 <akerr> has anyone used the cisco driver?
16:18:34 <asselin> jnrao, I'd start by looking at which tempest tests would cover the zone driver
16:18:40 <hemna_> akerr, not us yet.
16:18:44 <avishay> jgriffith: i'm surprised you're not volunteering to test FC :)
16:18:50 <jgriffith> avishay: :)
16:18:51 <xyang1> we have cisco switch
16:18:51 <jnrao> asselin :ok.
16:18:53 <hemna_> lol
16:19:00 <thingee> jgriffith, avishay: :)
16:19:12 * jungleboyj volunteers jgriffith
16:19:15 <jgriffith> avishay: won't be long before I have to I'm afraid
16:19:16 <xyang1> Cisco fixed a bug recently but there's still one more issue they are looking at
16:19:18 <hemna_> jnrao, attaches, detaches, live migration
16:19:22 <asselin> jnrao, then having ci run those tests is just configuration + tempest regex
16:19:26 * jgriffith slaps jungleboyj upside the head
16:19:31 <hemna_> jnrao, copy volume <--->image
16:19:32 <avishay> jgriffith: i'm sure
16:19:39 * jungleboyj blocks it with ninja like skills.
16:19:55 * jgriffith runs away
16:20:02 <thingee> OK :)
16:20:07 <flip214> May I suggest moving to the next point?
16:20:19 <avishay> if we get all storage drivers with CI by K that would be great
16:20:19 <jnrao> only attach and detach only for now.
16:20:25 <hemna_> avishay, +1
16:20:30 <akerr> flip214: +1, its almost lunch time
16:20:39 <avishay> FC zone drivers are bonus for now IMO
16:20:42 <flip214> akerr: no, the kids want dinner!
16:20:51 <thingee> How many zone manager drivers do we have at this point?
16:20:53 <thingee> hemna_: ^
16:20:55 <hemna_> thingee, 2
16:21:00 <hemna_> cisco, brocade
16:21:00 * bswartz gave up hope of eating lunch on wednesdays
16:21:05 <hemna_> but if we are doing CI
16:21:15 <hemna_> we need to do CI for the driver and the lookup services for each vendor
16:21:26 <jungleboyj> avishay: +1  That is the first goal.
16:21:37 <hemna_> each vendor's support with the FCZM is broken up into 2 components.   the driver and the lookup service.
16:21:38 <thingee> hemna_: ugh
16:21:45 <jnrao> hemna_:agree that we need CI for driver and lookup
16:21:50 <avishay> hemna_: is there zone driver-specific code in the storage driver?
16:21:56 <hemna_> avishay, no
16:21:58 <hemna_> well
16:22:06 <jnrao> avishay: no.
16:22:10 <jgriffith> wonder if we should take a look at how ZM was designed
16:22:13 <hemna_> there is a decorator on the initialize_connection and terminate_Connection methods for FC enabled drivers.
16:22:16 <jgriffith> think there's a better way
16:22:25 <akerr> is the interface into the lookup service not standard?  Can we claim compliance with both if we CI with only 1?
16:22:29 <hemna_> so if the FCZM is configured it gets called, w/o the volume manager being involved.
16:22:33 <thingee> perhaps we can reasonable and knowing that the zone manager works with one FC volume driver?
16:22:40 <jgriffith> ie turn it into a target, so to speak
16:22:44 <xyang1> jgriffith: sorry if I missed this earlier.  what happened to your FC driver?:)
16:22:46 <flip214> people, we're at article 2 of SEVEN on the list, with >⅓ of the time gone.
16:22:58 <jgriffith> xyang1: haven't written it yet :)
16:23:14 <hemna_> thingee, we can take this offline in the cinder channel if you like.
16:23:15 <avishay> do IBM, EMC, and HP use different ZM drivers in their CIs?
16:23:17 <jgriffith> xyang1: but it's relatively light I think
16:23:18 <hemna_> so we can move ahead ?
16:23:21 <thingee> hemna_: yes please
16:23:25 <thingee> jnrao: is that fine?
16:23:39 <hemna_> avishay, we just use 1 currently.  we don't have both vendor's switches at the moment.
16:23:42 <jgriffith> xyang1: no pun intended... FC == light :)
16:24:10 <thingee> #agreed nothing. need further discussion
16:24:10 <jungleboyj> avishay: We are still working on getting that working.
16:24:14 <thingee> #topic NFS/posix backup drivers again
16:24:15 <avishay> hemna_: i meant if IBM uses the brocade, and HP uses the cisco, for example, then at least both ZM drivers are exercised
16:24:22 <hemna_> avishay, true
16:24:23 <thingee> tbarron, kfox1111: hello
16:24:24 <jgriffith> avishay: +1
16:24:25 <tbarron> hi
16:24:25 <hemna_> avishay, gotcha
16:24:31 <kfox1111> hi
16:24:33 <thingee> alright lets all be nice here
16:24:43 <hemna_> avishay, I think xyang1 has cisco switches.  fwiw.
16:24:43 <xyang1> jgriffith: good, as long as you are planning to write one:)
16:24:46 <thingee> we have the posix backup driver
16:24:47 <thingee> #link https://review.openstack.org/#/c/82996/
16:24:59 <kfox1111> Yup. I believe we found a common solution.
16:25:00 <thingee> and the nfs backup driver
16:25:02 <thingee> #link
16:25:03 <tbarron> so Kevin and I have met and agreed on an approach that avoids duplication of functionality
16:25:11 <thingee> #link https://review.openstack.org/#/c/138234
16:25:15 <xyang1> hemna_, avishay: yes, we tested it.  there's one more issue they need to fix before we retest
16:25:24 <tbarron> I'm documenting here: https://review.openstack.org/#/c/130858/
16:25:30 <hemna_> jnrao, please join the #openstack-cinder channel to continue the FCZM discussion after the meeting
16:25:43 <kfox1111> Duncan recommended making the code from the swift driver more generic so it could be shared.
16:25:45 * DuncanT was very pleased to see the chunking base class
16:26:05 <thingee> #link https://review.openstack.org/#/c/139737/
16:26:14 <kfox1111> I've pulled most of the swift code out to a base class, and added parts of the posix driver to support big chunks.
16:26:23 <thingee> #agreed common solution of making swift driver more generic so it can be shared
16:26:36 <kfox1111> I've yet to port the posix driver to it, but I'm shooting for Friday. It should be a very small driver at that point.
16:26:39 <tbarron> +1
16:27:02 <thingee> ok, we might have to target to k-2 in review priority
16:27:03 <avishay> so the posix driver chunks the volume and stores the chunks in separate files?
16:27:05 <thingee> at this point
16:27:07 <tbarron> Then generic NFS driver - which needs to manage its own mounts - will inherit from posix
16:27:07 <kfox1111> The nfs driver can then build on it, doing the mount/config.
16:27:15 <thingee> is that fine?
16:27:30 <kfox1111> avishay: yes.
16:27:30 <jnrao> sorry lost connection.
16:27:40 <tbarron> We avoid duplication of function.
16:27:46 <kfox1111> thingee: sounds reasonable.
16:27:49 <avishay> sounds like a good plan
16:27:53 <thingee> alright great
16:27:56 <eharney> sounds good to me
16:27:56 <kfox1111> I am a little worried that swift changes may make merging hard.
16:28:01 <jnrao> thingee: CI For zone drivers: I will catch up with you later on this topic.
16:28:06 <thingee> kfox1111, tbarron: anything else?
16:28:09 <tbarron> avishay: yes
16:28:14 <kfox1111> if there arn't any for that driver, before then, it won't be a problem though.
16:28:19 <kfox1111> are there any schedualed?
16:28:28 * tbarron is a bit worried about delay since we are coupled to lots of changes
16:28:45 <thingee> backup drivers? I'm not sure. there are no more volume drivers accepted after k-1 is done
16:29:04 <tbarron> kfox1111: look at the multithreading for Swift backup driver change
16:29:22 <kfox1111> tbarron. ok. so at least one. :/
16:29:33 <tbarron> thingee: if we already have a review in process we're OK though, right?
16:29:35 <thingee> kfox1111: it's slow moving last I checked
16:29:53 <DuncanT> The multi-threading is proving a pain to get right
16:29:55 <thingee> tbarron: the requirement for k-1 just applies to volume drivers. not back up drivers
16:29:58 <tbarron> I'd like to get https://review.openstack.org/#/c/130858/ approved at this point (bp spec)
16:30:09 <tbarron> thingee: ack
16:30:12 <DuncanT> And probably wants to be multi-process to get significant wins
16:30:22 <thingee> tbarron: that's a good thing
16:30:24 <DuncanT> Since threading in python is painfully limitted
16:30:37 <tbarron> need DuncanT and eharney to look at the bp-spec
16:30:58 <tbarron> I will revise with kfox1111's comments from this morning
16:31:07 <thingee> ok I think this going a good direction
16:31:11 <thingee> #topic Discuss implementation direction of PureISCSIDriver CHAP support changes
16:31:17 <thingee> patrickeast: you're up
16:31:22 <tbarron> it has a plan of record for NFS, posix, and chunking
16:31:25 <thingee> #link https://review.openstack.org/#/c/137175/
16:31:27 <patrickeast> hi, so for https://review.openstack.org/#/c/137175/ there was some contention about the direction
16:31:29 <kfox1111> Thanks all. gtg.
16:31:36 <thingee> kfox1111: thank you
16:31:39 <patrickeast> and it was recommended to bring it up for discussion at the meeting
16:31:51 <patrickeast> the basic issue is that we don’t have a good place to store the CHAP secret
16:32:14 <patrickeast> the latest version of the changes use a new db table to store them
16:32:41 <patrickeast> i’ve put together a little bit more info and some answers to common questions here https://etherpad.openstack.org/p/pureiscsi-chap-faq
16:32:43 * thingee scans the change
16:33:15 <jgriffith> patrickeast: so first, I owe you for your patience and explaining the issue multiple times for me :)
16:33:35 <patrickeast> jgriffith: np
16:34:06 <hemna_> patrickeast, so FWIW, we had the same issue a while back
16:34:12 <jgriffith> patrickeast: Personally I don't mind the table entry, even the file is *ok* now that I understand the problem better.  So I'm good with whichever method the rest of the team deems better
16:34:25 <hemna_> we ended up finding a solution on our array to store the CHAP creds.
16:34:36 <patrickeast> hemna_: yea unfortunately we don’t have that ability
16:34:43 <patrickeast> hemna_: well, very easily
16:34:47 <eharney> i think it's credible that Cinder should store this info as a general idea
16:34:57 <hemna_> hehe patrickeast I didn't say ours was easy :P
16:35:05 <patrickeast> hemna_: lol
16:35:23 <hemna_> some drivers use cinder.conf
16:35:26 <hemna_> which has it's own issues
16:35:30 <jgriffith> So one thing we need to thnk about is sec people flip when this isn't stored encrypted
16:35:43 <patrickeast> yea :(
16:35:43 <jgriffith> We'll have to address that at some point... maybe later in K?
16:36:03 <eharney> i wonder if you can get around this issue a different way by working like LIO does
16:36:12 <jungleboyj> eharney: I was just waffling on that idea.
16:36:13 <avishay> maybe instead of another table, add a driver-specific column in the volume table, 255 char, driver can put whatever they want?
16:36:14 <hemna_> heh and if you put the encryption keys in cinder.conf, then how secure are we really?
16:36:20 <eharney> just generate them at attach time?
16:36:36 <jgriffith> avishay: interesting....
16:36:46 <jgriffith> I kinda like that... driver-metadata column
16:36:49 <hemna_> avishay, like....metadata :P
16:36:51 <jungleboyj> avishay: _1
16:36:52 <jgriffith> errr... table
16:36:56 <patrickeast> avishay: the problem is that we need to have access even when the volume table is potentially empty (sans the one being attached)
16:36:57 <jungleboyj> avishay: +1
16:37:00 <DuncanT> We already have two fields in the table for that
16:37:01 <avishay> seems overkill to add a new table for one driver
16:37:02 <avishay> hemna_: durrr, yea :)
16:37:03 <DuncanT> provider-*
16:37:06 <avishay> hemna_: admin metadata even
16:37:11 <jgriffith> patrickeast: it'd be an independent table
16:37:12 <rushiagr> oh metadata God :)
16:37:16 <patrickeast> oooh
16:37:17 <hemna_> avishay, yah that's what I was thinking.  admin metadata
16:37:35 <patrickeast> as long as it isn’t tied to a volume it will probably work
16:37:43 <patrickeast> thats been the biggest issue
16:37:44 <jgriffith> btw, what's up with the admin metadata that's there?
16:37:45 <hemna_> ?
16:38:00 <hemna_> patrickeast, but you need the CHAP to attach the volume.
16:38:02 <jgriffith> patrickeast: nope, wouldn't be tied to a vol at all
16:38:05 <hemna_> why can't it be tied to a volume ?
16:38:05 <eharney> is storing the chap credentials long term actually required?
16:38:18 <jgriffith> eharney: it is for some people :(
16:38:41 <patrickeast> so for our backend we don’t have volume specific credentials, they are specific to a host (more explaination in the FAQ i posted)
16:38:44 <eharney> is it?  you can't generate them at attach time and handle it dynamically then?
16:38:54 <hemna_> patrickeast, ours is the same way
16:39:04 <eharney> ahh
16:39:05 <thingee> eharney: that's what I was thinking
16:39:08 <jgriffith> eharney: trust me, that horse is dead and I beat it for a week :)
16:39:17 <thingee> jgriffith: ok :)
16:39:18 <DuncanT> Many drivers put in cinder.conf now - whatever alternative solution we go for needs to be more secure than that IMO to be worth changing - a field in the db is not inherently more secure than a database table
16:39:24 <hemna_> eharney, the problem is, you need credentials for the iscsi session to the host the first attach.
16:39:32 <DuncanT> s/than a database table/a conf file/
16:39:33 <hemna_> every attach after that needs the same credentials
16:39:52 <eharney> hemna_: yeah, different scheme, LIO does creds per-volume
16:40:04 <jgriffith> eharney: +1, same with SF
16:40:04 <eharney> got it
16:40:16 <jgriffith> makes life much easier
16:40:24 <patrickeast> DuncanT: yea i agree, the benefit is that you can have different credentials for each host instead of a globally used set of credentials
16:40:24 <ameade_> how far off would using something like barbican be?
16:40:32 <thingee> ok so I heard a new table and metadata.
16:40:35 <patrickeast> DuncanT: this was a feature specified by a customer
16:40:37 <jgriffith> ameade_: I probably wouldn't even consider it TBH
16:40:47 <thingee> metdadata won't work for patrickeast because that's tied to a volume
16:40:57 <jgriffith> thingee: I like the idea of a Vendor-Metadata table personally
16:41:04 <DuncanT> patrickeast: Customers specify all sorts of crazy things, I like to get the reasons *why* they want a thing
16:41:07 <jgriffith> thingee: and the idea of encrypting the keys in the db
16:41:09 <lpabon> ameade_: I agree, it should  be considerated
16:41:13 <hemna_> jgriffith, yah, we could have used that ages ago, if we had it.
16:41:31 <patrickeast> so i guess if we have buy-in for a new table we can hash out the details in the review/or in openstack-cinder, don’t want to take up too much of the meetin time
16:41:31 <thingee> #idea vendor-metadata table
16:41:35 <jungleboyj> Doesn't seem crazy
16:41:37 <DuncanT> vendor-metadata is what, one table per vendor?
16:41:41 <jgriffith> thingee: I would like to have a look/culling of all the metadata tables we have currently though
16:41:51 <jgriffith> seems we've become a bit "metadata happy"
16:41:52 <DuncanT> On record per vendor even?
16:41:59 <jgriffith> DuncanT: yeah
16:42:08 <jgriffith> DuncanT: err...no
16:42:12 * DuncanT votes for calling it other than meta-data
16:42:14 <hemna_> normalize it to a vendor table, and FK that to the metadata table?
16:42:14 <jgriffith> DuncanT: so one table, multiple entries
16:42:15 <thingee> patrickeast: I would like to take it offline to understand the point eharney made about getting the details on attach time. or point me to the answer on the faq
16:42:25 <hemna_> then you could have multiple entries in the table from different vendors
16:42:26 <thingee> but yes sounds like we have an approach to consider
16:42:34 <hemna_> thingee, +1
16:42:35 <jgriffith> columns like:  Vendor-Name, Vendor-ID, Driver....
16:42:39 <patrickeast> thingee: np, i’m happy to explain it in length ;)
16:42:40 <hemna_> jgriffith, yah
16:42:46 <jgriffith> and of course K/V's
16:43:12 <thingee> #topic Discuss database cleanup status and priorities
16:43:16 <thingee> e0ne: you're up
16:43:17 <e0ne> hi all
16:43:18 <jgriffith> my only concern is that maybe we have too much of that already.  And maybe admin meta is a good place for this particular problem?
16:43:23 <thingee> #link https://etherpad.openstack.org/p/cinder-kilo-db-cleanup
16:43:26 <jgriffith> Oh...gues we're done on that
16:43:26 <DuncanT> K/Vs make indexing near impossible though, so if there are going to be many entries we need something better
16:43:30 <e0ne> hi all. i don't want to take a lot of time
16:43:35 <e0ne> i just want to prioritize these things.
16:43:41 <e0ne> there are a lot of blueprints which could be outdated or important
16:43:43 <hemna_> jgriffith, isn't the admin meta data tied to a volume though ?
16:43:43 <jgriffith> DuncanT: FK in Volume column maybe?
16:43:58 <e0ne> also, there are no much perfermance impact from the db side when we'ge got 10000+ records and more then 1000 existing volumes
16:44:15 <jgriffith> hemna_: yeah :(
16:44:32 <e0ne> DuncanT: actually, indexes doesn't solve all problems
16:44:40 <e0ne> a lot of time takes to data transfer between db <> api <> client
16:45:11 <e0ne> e.g.  14ms for select in db bacome about 250 from python side:(
16:45:20 <DuncanT> e0ne: No, but a brute force of search of many K/V records is always worse
16:45:26 <avishay> e0ne: i've seen the same thing
16:45:56 <e0ne> DuncanT: i'll provide some profiler result with indexes
16:46:04 <e0ne> it will winn about 2-3ms:
16:46:38 <e0ne> we need to optimize data transfering too
16:46:49 <avishay> the most important thing IMO is to reduce the number of DB accesses.  we saw the osprofiler example with like 30 DB accesses for create_volume?
16:47:01 <DuncanT> e0ne: The win will totally depend on the number of records, number of K/Vs in each record etc
16:47:02 <e0ne> we select too many data when we don't need ir
16:47:04 <avishay> and storing heartbeats there, etc.
16:47:07 <e0ne> s/ir/it
16:47:15 <hemna_> avishay, +1
16:47:22 <jgriffith> avishay: +1, started cleaning some of those up... particularly update, followed by get :(
16:47:29 <e0ne> DuncanT: i tested with 100k+ records
16:47:41 <avishay> jgriffith: +1
16:48:39 <avishay> from what i've seen we've gotten rid of the O(N) operations (like one DB lookup per volume).  scale seems to be OK i think.
16:48:49 <e0ne> heartbits and deleted record is a really big problem on big env
16:49:01 <thingee> e0ne: so you're original question was to prioritize
16:49:15 <DuncanT> deleted records in quota reservations bit us recently
16:49:17 <thingee> your*
16:49:23 <DuncanT> 12 million of the damn things
16:49:29 <e0ne> thingee: yes. and verify existing blueprints
16:49:31 <hemna_> doh
16:50:04 <thingee> anyone want to work with e0ne on this?
16:50:08 <tbarron> DuncanT: :-)
16:50:17 <rushiagr> I'd like to volunteer
16:50:20 <rushiagr> myself
16:50:29 <e0ne> rushiagr: thanks!
16:50:45 <avishay> DuncanT: do they ever get deleted, or are they just marked as deleted?
16:50:49 <thingee> I'd recommend just carry conversations in #openstack-cinder and people might over read them and chime in.
16:50:56 <rushiagr> e0ne: :)
16:51:06 <thingee> #topic Discuss HA issues mentioned by Ops during the summit
16:51:10 <dulek> hi all
16:51:13 <dulek> #link https://etherpad.openstack.org/p/kilo-crossproject-ha-integration
16:51:15 <akerr> i've always wondered why deleted records stick around rather than getting deleted
16:51:15 <DuncanT> avishay: There's nothing in cinder to delete them at the moment, you need to implement something outside of cinder
16:51:22 <thingee> #link https://etherpad.openstack.org/p/kilo-crossproject-ha-integration
16:51:26 <dulek> this is from the summit session
16:51:46 <dulek> there are multiple cinder issues mentioned, one of them is no recovery in case of failure mid-task
16:51:54 <jgriffith> akerr: audit, billing, record-keeping etc etc
16:52:01 <dulek> our operators are constantly encountering this, so I'm motivated to work on it
16:52:17 <e0ne> dulek: +1 from me
16:52:18 <dulek> I can offer my full time + maybe another our developer to drive this topic through Kilo cycle, I would just need some guidance where and how to start
16:52:20 <jgriffith> dulek: curious, is anybody else in OpenStack solving this?
16:52:30 <avishay> DuncanT: gotcha thanks
16:52:48 <dulek> jgriffith: taskflow? Glance is integrating with in Kilo.
16:52:48 <thingee> dulek: great. i think there might be some discussions that happened at the summit that might overlap with this. I have not read this over myself though to verify atm
16:52:50 <jgriffith> dulek: ie is there any project that recovers from failure mid-action
16:52:56 <flip214> is the HA wanted on a per-service level?
16:52:57 <hemna_> jgriffith, wasn't one portion of this looking at removing the local locks in the volume manager?
16:53:02 <jgriffith> dulek: Glance is a trivial non-interesting case IMO
16:53:16 <flip214> because the easy solution is to put some services in a VM and have *that* HA via pacemaker...
16:53:26 <hemna_> jgriffith, and we had spoke about this in the meetup in paris to add 'ing' state checks in the API to prevent actions being taken
16:53:34 <dulek> flip214: Well, still you lose your request
16:53:59 <dulek> Also you have no possibility to safely close the service (for upgrade for example)
16:54:02 <jgriffith> flip214: that's not really the problem being discussed, it's more about rpc dropping on the floor mid-stream or db getting out of sync or stuck
16:54:11 <e0ne> flip214: we need some monitoring/healthcheck api for pacemaker
16:54:24 <jgriffith> flip214: solutions like pacemaker and A/A nodes are in existence
16:54:26 <dulek> I don't want to waste meeting time so maybe someone can volunteer to explain me status of that offline? ML replies were a little too vague for me to understand anything.
16:54:29 <thingee> dulek: I'll make an action item for myself to review this and see if some of the other stuff in progress takes care of the concerns. I'll provide a blueprint/spec when necessary.
16:54:49 <thingee> #action thingee to review HA etherpad to see what overlaps
16:54:56 <jgriffith> dulek: so for the sake of record keeping, I think it's a GREAT thing to work on
16:55:08 <jgriffith> dulek: just found the problems to be stated a bit too vaguely
16:55:19 <avishay> thingee: if you have any questions, i wrote most of the comments in the etherpad
16:55:29 <jungleboyj> jgriffith:  +1
16:55:29 <thingee> avishay: thanks!
16:55:41 <dulek> thingee: Can you answer on the ML then when you finish?
16:55:51 <dulek> e0ne: you also wanted to kickoff a HA-related topic today, right?
16:55:59 <thingee> jgriffith: usually that's how it is. I think it's important to point dulek to the work in progress that are smaller chunks to work towards this
16:56:13 <thingee> dulek: sure
16:56:17 <dulek> thingee: +1, that's what I need to start helping. :)
16:56:26 <e0ne> dulek: i'm ready to start it almost anytime:)
16:56:33 <thingee> dulek: ok I need to move onto the next topic, sorry
16:56:38 <jgriffith> thingee: yeah, I'd appreciate that pointer as well :)  Maybe link them on the etherpad
16:56:47 <thingee> #topic  Third Party CI updates
16:56:50 <thingee> asselin: go go go!
16:56:59 * jgriffith keeps loosing track of thangp and harlowja_away 's work :(
16:57:03 <asselin> hi, real quick:
16:57:05 <asselin> Would like to invite Cinder third party ci "operators" to third party ci meeting: https://wiki.openstack.org/wiki/Meetings/ThirdParty
16:57:09 <thingee> #link https://wiki.openstack.org/wiki/Meetings/ThirdParty
16:57:19 <asselin> would be great to have more participation
16:57:27 <asselin> Meeting time change is proposed on mailing list. Vote here: http://lists.openstack.org/pipermail/openstack-dev/2014-December/052546.html
16:57:33 <asselin> go vote for new meeting time!
16:57:43 <thingee> #link http://lists.openstack.org/pipermail/openstack-dev/2014-December/052546.html
16:57:46 <asselin> lastly, Asselin is proposing an in-tree 3rd party ci solution. Idea well received in 3rd party ci & infra meetings. Please review spec and/or let me know your interest.
16:57:57 <asselin> #link Gerrit topic "thirdpartyci" query: https://review.openstack.org/#/q/topic:thirdpartyci,n,z
16:58:49 <eikke> asselin: we're interested in this
16:59:20 <flip214> asselin: are the times UTC?
16:59:22 <asselin> you can ping me in the -cinder channel, or comment on spec.
16:59:26 <asselin> flip214, yes
16:59:34 <flip214> +1 for A
16:59:34 <asselin> eikke, great :)
16:59:35 <rushiagr> /whois asselin
16:59:55 <asselin> rushiagr, Ramy Asselin
17:00:00 <jungleboyj> :-)
17:00:05 <jgriffith> rushiagr: who is John Galt!
17:00:13 <flip214> that's a rather manual way of answering whois.
17:00:16 <thingee> thanks everyone!
17:00:19 <thingee> #endmeeting