16:01:13 <jgriffith> #startmeeting cinder
16:01:14 <openstack> Meeting started Wed Feb  5 16:01:13 2014 UTC and is due to finish in 60 minutes.  The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:17 <openstack> The meeting name has been set to 'cinder'
16:01:19 <rushiagr> o/
16:01:23 <bswartz> hi
16:01:23 <avishay> hello
16:01:25 <jgriffith> hola everyone.. it's been a while :)
16:01:32 <glenng> Hola
16:01:36 <jgriffith> seems Wed is the best day of week to fly :)
16:01:43 <avishay> :)
16:01:58 <dosaboy> howdi
16:02:09 <DuncanT> hey
16:02:14 <jgriffith> dosaboy: and DuncanT both!!!
16:02:17 <jgriffith> W00T
16:02:28 * DuncanT gets nervious
16:02:33 <jgriffith> Nahh
16:02:34 <dosaboy> d-ream team
16:02:47 <jgriffith> alright... let's fire this thing up
16:02:48 <xyang> hi
16:03:00 <jgriffith> #topic I3 Status
16:03:14 <jgriffith> https://launchpad.net/cinder/+milestone/icehouse-3
16:03:26 <jgriffith> A number of folks snuck some things in over the week-end
16:03:39 <jgriffith> New driver submissions from prophet-store and others
16:04:09 <jgriffith> basically we're at submission freeze, so if you BP isn't in it'll need an exception after this point
16:04:37 <dosaboy> o/
16:04:37 <jgriffith> The way it's *supposed* to work is that reviewers would come across somethnig that isn't approved and not approve it in review
16:05:24 <avishay> jgriffith: everything that is approved is marked as such?
16:05:39 <jungleboyj> Howdy all.  Sorry I am late.
16:05:52 <jgriffith> avishay: not yet :(
16:06:03 <jgriffith> avishay: but I'll have it updated later today
16:06:12 <avishay> jgriffith: ok cool
16:06:17 <jgriffith> avishay: and the big thing is I'd like to NOT see people sneaking in new stuff
16:06:27 <jgriffith> if somebody has a bp they need to add let me know
16:06:39 <jgriffith> don't just add it and mark it as targetted
16:06:50 <avishay> i would never do such a thing
16:06:56 * avishay whistles
16:06:58 <jgriffith> of course none of the people that show up to this meeting do that so no point in me mentioning it here
16:07:06 <dosaboy> jgriffith: I've had 1137908 on that list a while and i've decided to break it into 4 patches to make it more review-friendly
16:07:12 <dosaboy> jgriffith: should be done buy end of week
16:07:26 <jungleboyj> avishay: :-)
16:07:27 <jgriffith> dosaboy: great, you were next on my list of topics :)
16:07:31 <dosaboy> unless anyone objects
16:07:34 <coolsvap> jgriffith: I would like to propose a new driver bp
16:08:17 <dosaboy> reckon it makes sense cause it got kinda bloated
16:08:34 <jgriffith> dosaboy: sounds good to me
16:08:41 <avishay> jgriffith: is the cert script a necessary precondition for +2ing drivers?
16:08:43 <coolsvap> jgriffith: https://blueprints.launchpad.net/cinder/+spec/astute-nfs, we are mostly on completion with this
16:08:51 <jgriffith> coolsvap: I think you're going to be late on that
16:09:02 <jgriffith> coolsvap: we want things in the queue by end of next week
16:09:19 <jgriffith> coolsvap: but if you've got one to propose get the BP posted this morning
16:09:34 <jgriffith> coolsvap: I thought we already talked about this last week and you said the BP was on it's way?
16:09:51 <jgriffith> avishay: I'd like to make it as such
16:10:06 <jgriffith> avishay: I've got new changes in tempest and devstack that should fix things up
16:10:19 <jgriffith> avishay: and hema ran into something weird that he patched yesterday
16:10:30 <jgriffith> We need a place to put the files though
16:10:45 <jgriffith> That part I don't have...
16:11:10 <jgriffith> but maybe I could suggest google-docs, or maybe give a lnk to my S3 container?
16:11:13 <avishay> jgriffith: from what i remember there is the IBM NAS driver and 2 HP drivers - so no merge until a successful run?
16:11:19 <coolsvap> jgriffith: the blueprint I have already added, the code can be pushed by friday at the earliest
16:11:26 <thingee> o/
16:11:30 <avishay> thingee: yo
16:11:31 <jgriffith> coolsvap: then you're fine
16:11:31 <xyang> jgriffith: if there are still bugs in the cert tests, it is hard to make it a requirement for new drivers
16:11:51 <jgriffith> xyang: there aren't bugs in the cert test actually
16:12:13 <jgriffith> xyang: the issues were setting protocol and vendor name in tempest
16:12:41 <jgriffith> the devstack change: https://review.openstack.org/#/c/68726/
16:12:47 <jgriffith> is simply to make life "easier"
16:13:05 <jgriffith> xyang: the script should be functional, and if it's not please let me know
16:13:18 <avishay> jgriffith: can you make some small guide to running it?  it seems that running unmerged drivers requires changing the script?
16:13:19 <jgriffith> xyang: it's been out there for a number of weeks, I would've hoped people would've tried it
16:13:39 <jgriffith> avishay: yes, that's an excellent point
16:13:47 <jgriffith> avishay: I'll get something documented and sent out
16:13:54 <thingee> coolsvap: for this new driver, I would also like to see the cert tests passing ^
16:13:56 <avishay> jgriffith: OK thanks
16:13:57 <jgriffith> avishay: or just update cinder's dev guide
16:14:21 <avishay> jgriffith: as far as i'm concerned it can be comments at the top of the script :)
16:14:27 <xyang> jgriffith: sure, I haven't got chance to try it yet, but will give a try.  I hope we can at least do that after 2/18?
16:14:32 <coolsvap> thingee: sure I have that on the checklist
16:14:56 <jgriffith> avishay: yeah... I was just thinking "who knows how long it will take till a change lands" :)
16:15:19 <avishay> jgriffith: will the recent fixes help with the errors Nilesh reported for the IBM NAS driver?
16:15:26 <jgriffith> I guess I'm not sure why not just run it
16:15:35 <avishay> OK
16:15:35 <jgriffith> It only takes around 5 minutes
16:15:47 <avishay> fair nuff
16:15:51 <jgriffith> avishay: I'm not sure what errors he reported
16:15:51 <jungleboyj> :-)
16:16:00 <jgriffith> avishay: sorry... that was towards xyang 's question :)
16:16:03 <jungleboyj> jgriffith: He was seeing like 4 failures.
16:16:22 <jgriffith> jungleboyj: I'll look, but I suspect it was the vendor_id and protocol
16:16:31 * jungleboyj still needs to go review the code again.
16:16:33 <jgriffith> jungleboyj: but I thought that was *2* errors not 4
16:16:54 <jgriffith> so there's a api.volume.test_volumes_types
16:17:03 <jgriffith> it sets volume type and extra-specs
16:17:22 <avishay> timeout in upload_image
16:17:22 <jgriffith> the default in the test is vendor = OpenSource and protocol=iSCSI
16:17:39 <jgriffith> avishay: hmmmm... might be a bug in his driver :)
16:17:43 <akerr> avishay: we see upload_image timeout with Netapp drivers too
16:17:52 <jgriffith> akerr: NFS version?
16:18:00 <jungleboyj> Ruh roh.
16:18:04 <akerr> jgriffith: I think 4?
16:18:06 <avishay> jgriffith: he claims it works manually and uploaded logs
16:18:11 <avishay> jgriffith: but maybe
16:18:18 <xyang> jgriffith: I'll try sooner, but if we run into issues, I don't know how long it will take for it to complete successfully.
16:18:31 <jgriffith> xyang: fair enough
16:18:46 <jgriffith> avishay: akerr so ping me after meeting and I can help debug
16:18:48 <sneelakantan> jgriffith: I ran the script for vmware driver, and had issues with 196 sec timeout during upload to image too
16:18:53 <avishay> jgriffith: sure
16:19:05 <sneelakantan> jgriffith: I set the timeout higher and it passed
16:19:36 <jgriffith> since we don't gate on NFS and it's become evident most people don't run the tests I wouldn't be surprised if there's a bug in the driver or in the tempest test
16:19:43 <jgriffith> sneelakantan: ahhh
16:19:53 <jgriffith> sneelakantan: so its just painfully slooooowwwwww
16:20:21 <sneelakantan> jgriffith: yep. Take about 4 mins to upload a 1GB file. May be just my env.
16:20:34 <jgriffith> holy cow... that's horrid
16:20:35 <bswartz> not all of us has all-flash storage arrays
16:20:38 <bswartz> lol
16:20:45 <avishay> haha
16:20:46 <jgriffith> bswartz: LOL...
16:20:56 <bswartz> 4 mins for 1 GB does seem a little off though
16:20:58 <jgriffith> bswartz: I didn't know you used stone tablets and chissels though
16:21:17 <jungleboyj> Isn't everyone one running on little bogged down VMs?
16:21:21 * jgriffith should've changed the topic
16:21:27 <jgriffith> #topic cert test
16:21:44 <jgriffith> Rewind about 5 minutes for beginning of topic :)
16:22:22 <jungleboyj> :-)
16:23:11 <jgriffith> Ok... guess that's that :)
16:23:22 <dosaboy> while we are on the topic of uploads, were there not plans a while back to refactor the cinder.image?
16:23:24 <jgriffith> #action jgriffith to updte docs on how to use cert-test
16:23:35 <dosaboy> since we now have a kludge of v1 and v2
16:23:56 <jgriffith> dosaboy: more importantly we're still not using v2 as default
16:24:03 <dosaboy> agreed
16:24:07 <jgriffith> dosaboy: I think there are few things we need to get fixed here
16:24:27 <DuncanT> v2 still isn't in use in many / most places as of the last summit
16:24:37 <thingee> jgriffith: v2 is routed by default
16:24:49 <dosaboy> DuncanT: um i seem to remember hearing at the summit that they are planning to deprecate v1
16:24:58 <thingee> dosaboy: yes
16:25:25 <jgriffith> thingee: I think someone pointed out an issue last week on that
16:25:32 <jgriffith> thingee: I'll have a look after the meeting
16:25:49 <kmartin> jgriffith: we found a few problems in the 3PAR drivers by running the cert test, hemna will be posting a patch soon to fix them
16:25:57 <thingee> hm, news to me =/
16:25:58 <jgriffith> kmartin: nice!
16:26:11 <jgriffith> thingee: I could be mistaken, I"ll go back through my notes
16:26:21 * jgriffith keeps notes
16:26:45 <jgriffith> dosaboy: so to your point/questin
16:26:48 <jgriffith> question
16:26:56 <jgriffith> We talked about that, we should look at it
16:27:05 <thingee> I'm not sure I understand.
16:27:09 <thingee> v2 is routed
16:27:16 <jgriffith> Honestly I was hoping to get some more improvements in image.py
16:27:19 <thingee> it's just not warning your about using v1 still
16:27:32 <jungleboyj> thingee: What do you mean routed?
16:27:42 <thingee> the v2 controllers are routed
16:28:06 <thingee> if you run G, you get v2
16:28:15 <thingee> you just have to make sure keystone catalog stuff is setup properly
16:28:29 <dosaboy> jgriffith: i'm happy to help out with glance-related work if I find the time
16:28:54 <jungleboyj> thingee: From what I have seen, you can't count on Keystone being set up properly.
16:29:38 <jgriffith> dosaboy: thanks
16:29:59 <thingee> jungleboyj: people can setup anything wrong. how is this different?
16:30:04 <jgriffith> thingee: deep breaths, it's ok... lemme see if I can find out what the complaint was
16:30:07 <thingee> you can put v5 when you meant v1
16:30:10 <jgriffith> thingee: perhaps it's nothing
16:30:23 <jungleboyj> thingee: True enough.
16:30:46 <thingee> I guess instructing with a deprecation warning that v1 is old, use v2 would help
16:30:57 <thingee> do we want to do that in I?
16:31:21 <jgriffith> thingee: it would probably be good if we did
16:31:26 <thingee> seems like the next step
16:31:27 <thingee> ok
16:31:28 <jungleboyj> It also seems we keep adding to V1 so people don't have the incentive to move foward.
16:31:29 <jgriffith> else nobody will move
16:31:37 <jungleboyj> jgriffith: +2
16:31:44 <thingee> #action thingee will deprecate v1
16:31:57 <jgriffith> jungleboyj: well, we're supposed to be rejecting feature adds to V1 already :)
16:32:00 <thingee> jungleboyj: +1
16:32:18 <thingee> yes, I have to ask that everyone be more strict on accepting patches where people add to v1.
16:32:23 <jungleboyj> jgriffith: That was why I was pushing back against the multi volume change.  :-)
16:32:39 <jgriffith> jungleboyj: indeed :)
16:32:41 <thingee> the only exception in my opinion is having cinderclient v1 expose something v1 already have...imo
16:32:53 <thingee> I have seen those changes come lately
16:32:54 <jgriffith> thingee: agreed
16:33:11 <jgriffith> Ok... we've digressed a bit, but at least on something relevant
16:33:22 <coolsvap> jgriffith jungleboyj thingee just to confirm, should multi volume create go in v1 or not?
16:33:23 <jgriffith> Let's see what bswartz has in store for us this am
16:33:32 <jgriffith> #topic multiple pools per backend
16:33:33 <bswartz> hah
16:33:46 <bswartz> well the short answer is -- I tried to make this work and I couldn't
16:34:03 <bswartz> I think winston was right that this requires a DB change or something more invasive than what i had in mind
16:34:24 <jungleboyj> coolsvap: I would say no based on the discussion we just had.
16:34:46 <bswartz> this is because of how the scheduler tracks free space -- if you try to have multiple pools inside a backend, but volumes aren't associated with them in the DB, then you can't figure out allocated space
16:35:23 <avishay> i think multi-create should go into v1 - works with v1 APIs
16:35:24 <bswartz> so I'm going to look at other approaches to solving this problem, because simply manding multiuple backends (one per pool) doesn't seem like the right approach either
16:35:39 <avishay> bswartz: interesting
16:35:40 <jgriffith> avishay: coolsvap let's hold off on that for now
16:36:04 <bswartz> I may propose a DB change for J to actuall allow cinder to track multiple pools
16:36:06 <bswartz> I'd like other ideas though
16:37:07 <bswartz> the goal is to allow 1 array to present multiple pools of free space (each with it's own set of capabilities) to the scheduler without forking lots of c-vol processes
16:37:12 <jgriffith> bswartz: so going the other way and chaging how the scheduler interprets space isn't going to cut it
16:37:16 <bswartz> because it's dumb to have a bunch of processing managing 1 array
16:37:36 <jgriffith> bswartz: so something that came up before was an aggregated backend
16:37:50 <jgriffith> bswartz: one interface, multiple backends
16:38:04 <bswartz> jgriffith: any docs or BPs on that?
16:38:06 <jgriffith> the multiple backends abstracted into a single cinder driver interface
16:38:16 <jgriffith> bswartz: no, I just talked about it in Hong Kong
16:38:25 <bswartz> jgriffith: yes that's another reasonable approach to implement what I want
16:38:37 <DuncanT> bswartz: processes are cheap. If the config file is sane, who cares if we have 1 or 100?
16:38:39 <jgriffith> bswartz: but admittedly everybody was more interested in my 'report stats' was wrong
16:38:45 <avishay> jgriffith: a pseudo-driver that calls other driver instances?
16:39:03 <bswartz> DuncanT: it's more of an issue if htey have to be in sync with eachother for some reason
16:39:11 <jgriffith> avishay: well... it should be moore sophisticatd than that
16:39:22 <jgriffith> avishay: but at a high level, yes
16:39:32 <avishay> DuncanT: i know one issue people are seeing is lots of SSH connections to arrays because each driver instance has its own - sometimes you can run out of connections
16:39:37 <avishay> jgriffith: gotcha
16:40:00 <DuncanT> bswartz: avishay: Fair enough. Just wanted to shoot down the straw man and find out what the real problem was
16:40:05 * coolsvap was working on a poc similar to jgriffith's
16:40:07 <jgriffith> ssh, xml, smis... three acronyms that should be banned :)
16:40:27 <bswartz> I think we want to keep a single DRIVER per array, because there's some state you wouldn't want to have to keep multiple copies off, but somehow that needs to appear to teh scheduler/manager as multiple things
16:40:44 <jgriffith> bswartz: coolsvap maybe we sould spend some time together and see if we can arrive at a good solution for this?
16:40:46 <avishay> jgriffith: :)
16:41:02 <jgriffith> sort of a "parent-driver" approach
16:41:12 <bswartz> yes, but I don't see it happening in icehouse sadly
16:41:26 <jgriffith> bswartz: yeah, time is not our friend
16:41:43 <jgriffith> bswartz: but I don't know that any other proposal would help there either
16:41:57 <jgriffith> bswartz: unless you have a working proposal that you want to run with
16:42:12 <bswartz> no I think we need to start thinking about what we'd like to see in J
16:42:18 <coolsvap> jgriffith: sure
16:42:21 <jgriffith> bswartz: fair
16:42:24 <bswartz> icehouse will have to ship without support for this because I don't have any bright ideas
16:42:24 <jgriffith> ok
16:42:29 <eharney> bswartz: i also have some interest in this as i've been thinking about issues leading me toward similar ideas
16:42:29 <jgriffith> well thanks for the update
16:42:46 <jgriffith> the "NO DB change" edict is going to make things difficult
16:42:59 <bswartz> and I don't want to half-ass this either, I think it's important to get right
16:43:17 <avishay> bswartz: sounds good
16:44:04 <jgriffith> alright, we def have multiple parties interested in this
16:44:36 <jgriffith> Let's synch up later this week/early next week and get all of us on the same page
16:44:53 <bswartz> winston was interested too -- I suspect here's not here today
16:44:56 <jgriffith> It might be J timeframe but it would help if we went in to J with a plan
16:45:01 <jgriffith> or at least all on the same pge
16:45:02 <jgriffith> page
16:45:11 <bswartz> s/here's/he's/
16:45:23 <jgriffith> #topic open-discussion
16:45:30 <thingee> oh
16:45:31 <dosaboy> is the cinder hackathon going ahead?
16:45:33 <thingee> hackathon
16:45:37 <dosaboy> :)
16:45:37 <thingee> dosaboy: ;)
16:45:38 <avishay> hah
16:45:40 <thingee> haha
16:46:04 <avishay> i'm in, remotely
16:46:09 <dosaboy> i was gonna offer python-mock help where possible
16:46:14 <jungleboyj> I should be in remotely.
16:46:36 <thingee> so I wanted to try something a bit different. it seems like remote is the way to go with short notice.
16:46:45 <rushiagr> I will be too, remotely
16:47:01 <thingee> but the different thing to try is something like google hangout, team speak or something where communication is shown as us being a bit more focused
16:47:04 <coolsvap> me too remote
16:47:09 <thingee> do people think that would help?
16:47:12 <thingee> stupid idea?
16:47:20 <jgriffith> thingee: good idea IMO
16:47:25 <jgriffith> :)
16:47:28 <DuncanT> Certainly worth a try
16:47:31 <avishay> never tried either of those but worth a try
16:47:42 <DuncanT> Some of the teams here use google hangouts for such things
16:47:49 <coolsvap> yes we tried hangout at solum earlier
16:47:52 <thingee> there is a limit on hang out though right?
16:48:03 <rushiagr> yes I guess. 10 people
16:48:04 <dosaboy> 10 afaik
16:48:16 <dosaboy> but you can add on with call ins
16:48:28 <dosaboy> dial ins even
16:48:42 <rushiagr> or have two guys sit in front of one laptop :D
16:48:48 <thingee> that works. we don't really need to see faces :)...I don't think anyone wants to be staring at me all day making weird faces as I do reviews
16:49:02 * jungleboyj laughs
16:49:08 * jgriffith knows you don't want to see his mug all day
16:49:19 <thingee> jgriffith: just point it at the dog all day
16:49:25 <avishay> haha
16:49:29 <jgriffith> thingee: now there's an idea
16:49:30 <jungleboyj> You guys would have to deal with me headbanging and dancing in my chair.  ;-)
16:50:14 <thingee> ok, lets try something like that. so plan on having some headset ready
16:50:22 <avishay> thingee: it seemed you had a basic agenda - do you want to make a more detailed list of things to tackle?  how will it run?
16:50:36 <thingee> feb 24-26 sound good still?
16:50:57 <jungleboyj> thingee: So what is the idea?  We will talk through reviews quickly and interactively.  Talk out the solutions to avoid multiple patches?
16:50:59 <avishay> ok for me
16:51:16 <jungleboyj> avishay: Jinx.  :-)
16:51:20 <avishay> is it for reviews or for fixes, or both?
16:51:29 <thingee> avishay: so jgriffith had the idea of leaving it open. I just want the focus around getting patches through and some of the things I mentioned fix
16:51:33 <thingee> stability/bug fixes
16:52:12 <avishay> so kind of like a 3-day bug squashing event with voice/video
16:52:19 <jungleboyj> 24th - 26th still looks good to me.
16:52:29 <thingee> it might be most of the time it's people muted, but it's a way to quickly get attention and people talking out complicated things.
16:52:33 <dosaboy> who is sponsoring the beer tent?
16:52:40 <avishay> works for me
16:52:45 <jungleboyj> dosaboy: Good question!
16:52:48 * DuncanT wonders if there will be many on in a GMT timezone, or close? Avishay I guess...
16:52:49 <jgriffith> thingee: I did?
16:52:57 <thingee> it's an experiment. no promises it'll be perfect
16:53:10 <thingee> and yes video if you don't mind people seeing you :)
16:53:34 <thingee> Can I just get an idea of who is interested.
16:53:39 <thingee> just a rough count
16:53:45 <jgriffith> 0/
16:53:48 <avishay> me
16:53:50 <DuncanT> O/
16:53:51 <eharney> me
16:53:55 <coolsvap> +1
16:53:55 <jungleboyj> me
16:54:06 <dosaboy> o/
16:54:25 <jgriffith> I think we can count winston as well
16:54:29 <thingee> great we might have room for google hangout or something.
16:54:31 <thingee> yea
16:54:38 <jungleboyj> jgriffith: +1
16:54:39 <jgriffith> and likely harlowja_away
16:54:49 <coolsvap> yes winston-d ++
16:54:54 <jungleboyj> winston mentioned being in.
16:54:54 <jgriffith> eharney: not interested?
16:55:04 <eharney> i jumped in above ^  :)
16:55:12 <jgriffith> eharney: DOH!
16:55:13 <jgriffith> my bad
16:55:26 <jgriffith> I'd like to see all core members participate
16:55:27 <thingee> I'll start an etherpad and people can list bug links they're interested on working on. you also don't have to work on bugs, you could be finishing a bp or doing a lot of reviews
16:55:30 <jungleboyj> eharney: we used the wrong notation.
16:55:33 <jgriffith> personally
16:55:33 <jungleboyj> eharney: _1
16:55:39 <thingee> most importantly, it's three days that we're going to focus
16:55:42 <jgriffith> unless they have scheduling conflicts of course
16:55:50 <thingee> I'm planning to take the time off from work to really focus
16:56:13 <thingee> nobody is expected to do that of course, but the idea is just dedicated time to really focus on getting things through review and stability
16:56:23 <jungleboyj> thingee: What hours are we talking?
16:56:30 <thingee> good question
16:56:31 <avishay> hopefully gate will cooperate
16:56:49 <jungleboyj> thingee: I should block my calendar accordingly.
16:56:52 <avishay> (i just jinxed it, didn't i)
16:57:08 * jungleboyj looks sternly at avishay
16:57:56 <avishay> jungleboyj: i blocked my calendar from 8AM to 10PM, because who want boring calls and meetings anyway
16:58:07 <jungleboyj> avishay: :-)
16:58:17 <thingee> so I know avishay is going to have different TZ. winston will be in the US that time
16:58:35 <thingee> is there anyone else outside the US?
16:58:41 <coolsvap> I will have to take off from daytime work
16:59:17 <thingee> ok, we're running low on time. we can take the TZ talk to #openstack-cinder
16:59:23 <kmartin> I will let hemna know that he has to participate
16:59:32 <thingee> kmartin:  :)
17:00:17 <avishay> cool, we're out of time
17:00:26 <jgriffith> thanks everybody
17:00:31 <jungleboyj> Wow, that was fast.
17:00:42 <jgriffith> most of us are all here all day in #openstack-cinder so feel free to ping
17:00:47 <jgriffith> #endmeeting