16:03:40 #startmeeting cinder 16:03:41 Meeting started Wed Jul 3 16:03:40 2013 UTC. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:03:42 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:03:44 The meeting name has been set to 'cinder' 16:03:50 o/ 16:03:52 haha... starmeeting... startmeeting 16:03:55 figure it out virtbot! 16:03:55 \o 16:03:59 hi 16:04:05 hi again 16:04:08 hi 16:04:13 hehe 16:04:19 hlo 16:04:27 med_: yo! 16:04:31 kk... 16:04:34 let's roll 16:04:48 #topic update on ceph backup 16:04:51 dosaboy: ?? 16:04:57 o/ 16:04:59 ok so 16:05:09 first two parts of bp merged 16:05:21 thanks all for really good feedback and reviews 16:05:35 final part i.e. diff backups is WIP and almost ready 16:05:47 https://blueprints.launchpad.net/cinder/+spec/cinder-backup-to-ceph 16:05:53 I need to think of a decent way to do incremental backups with what we have 16:06:03 i might create a separate task for incremental backup since current task is for differential backups 16:06:16 also needs more testing 16:06:28 and happy to see mkoderer stress testing :) 16:06:35 so keen to hear more about that 16:06:35 yesss ;) 16:06:44 question for y'all 16:06:57 do we have a place for functional tests? 16:07:05 dosaboy: tempest... 16:07:14 ok 16:07:17 dosaboy: but that's not appropro here 16:07:33 dosaboy: that's what's used for gate tests, so we would need to look at that 16:07:40 right 16:07:42 dosaboy: ie having tests in tempest that aren't gated 16:07:44 I guess the goal is 100% unit test coverage :-) 16:08:01 matel: :) 16:08:02 matel: that works already ;) 16:08:03 there are a lot of corner cases with this code 16:08:14 and not all reachable with unit tests 16:08:27 so I think accompanying func tests would be good 16:08:31 I have been doing some 16:08:39 but nothing commited 16:08:45 functional that is 16:08:50 dosaboy: ideally... you could deploy devstack 16:08:55 so yeah thats kind of it 16:08:59 dosaboy: configure with your gear and run tempest 16:09:03 jgriffith: that is what I am using 16:09:14 I am testing with devstack and two ceph clusters 16:09:15 right... but what I'm saying is 16:09:28 oh yeah 16:09:28 the whole point of the API's is to abstract what those back-ends are 16:09:43 so you should be able to write generic functional tests to work across the board 16:09:52 yeah so I have tested compatiblity with others drivers and backends 16:09:57 the gates and official stack testsing will continue to use swift 16:10:02 e.g. rbd drver with swift backup 16:10:10 lvm driver with ceph backups etc 16:10:30 dosaboy: right, so that matrix is a bit tricky to support in the OpenStack CI obviously :) 16:10:33 is the question just where do we check in functional tests? 16:10:34 I *hope* I have covered most variants 16:10:39 guitarzan: yes 16:10:45 dosaboy: I am quite sure we will test all Ceph cases here 16:10:55 but no lvm 16:10:57 mkoderer: excellent! 16:11:01 guitarzan: but there's also a question IMO about how device specific those functional tests are 16:11:27 well from the api perspective they are device agnostic 16:11:44 For specific devices (other than ref impl) I'd just suggest a github repo of your own that people can use/contribute to 16:11:44 and tests would have to be repeated for each device type 16:11:50 non-openstack official 16:11:50 is there a problem with including device specific test scenarios? 16:12:00 jgriffith: sounds like a plan 16:12:11 guitarzan: there is if you gate on them 16:12:20 ok well that's it from me 16:12:21 jgriffith: i guess device specific functional tests should be done by device vendors since they have the *device* 16:12:24 as no one may have the devices 16:12:32 are the gate tests in the cinder tree? 16:12:39 guitarzan: no 16:12:45 guitarzan: they're in tempest 16:13:15 glance has functional tests for backends that are disabled when the backend isn't setup 16:13:15 guitarzan: are you proposing that we open up the cinder tree for device specific functional tests? 16:13:25 guitarzan: I'd rather not do that unless everybody else disagrees 16:13:29 jgriffith: I'm tossing that option out there, yes 16:13:34 * guitarzan shrugs 16:13:51 I can't see people managing/maintaining and reviewing code for devices they know nothing about and can't run 16:14:04 they already do that for driver code 16:14:11 guitarzan: Ok 16:14:25 guitarzan: go for it then 16:14:27 fine by me 16:14:55 jdurgin1: those live in the glance tree? 16:15:16 yes 16:15:23 so everybody agree with guitarzan that folks should just put their vendor specific functional tests in the cinder trunk and we should review maintain said code? 16:15:36 * dosaboy looks at glance tree 16:15:37 * guitarzan chuckles 16:15:46 guitarzan: why chuckle? 16:15:58 can it be separated from the always-run unit tests? different folder or something? 16:16:07 eharney: I think that's the idea 16:16:07 eharney: yes 16:16:21 jdurgin1: glance functional tests had some refactoring in week ago (last week?) 16:16:24 why would we want funcitonal tests in the cinder tree? aren't unit tests enough? 16:16:36 Unit tests are never enough 16:16:40 not usually 16:16:49 bswartz: there are always going to be cases that a unit test cannot cover 16:16:51 i'm not sure i see the value of tests that most people will never run though 16:16:53 bswartz, well, as noted, functional tests test differently. 16:17:00 eharney: +1000 16:17:10 eharney: that's a good point 16:17:32 it makes sense for open source backends... 16:17:39 yeah but the purpose of the unit tests aren't to prove the driver is correct, they are to catch idiotic mistakes from breaking drivers by accident 16:17:47 jgriffith, are you concerned about bitrot cruft, or just the initial review time/mindshare it takes to get in? 16:17:53 functional tests are valuable, but I don't see why they need to be in the cinder tree 16:17:55 med_: both 16:17:56 i think where test code sit in doesn't matter, whether tests being done regularly is more important. 16:18:02 med_: just don't see much value 16:18:09 med_: ROI is not very good 16:18:19 jdurgin1: do you know when and how the glance func tests are run? 16:18:29 like what do they run against 16:18:34 are they automated 16:18:35 vendors should have QA team publish test reports/file bugs against their driver regularly. 16:18:55 dosaboy: they're not automated afaik 16:19:05 med_: that and maintaining. driver maintenance has been a problem too 16:19:08 winston-d: +1 16:19:11 winston-d: I still say that's simple, just run tempest with your dev configured at a minimum 16:19:12 glance has them as an option in run_tests 16:19:42 jgriffith: but i don't have a solidfire or 3par 16:19:52 winston-d: and there's my point :) 16:20:11 If only the the vendor can run them while keep them in cinder, let the vendor keep them somewhere. It will also increase review times 16:20:14 winston-d: so why would we put solidfire or 3par specific functional tests in the cinder tree 16:20:30 winston-d: why not just let folks use tempest on their own and configure what they want to test 16:20:33 i think we should be worried about adding yet more code that no one will really know anything about 16:20:40 i have to agree w/ kmartin there. The maintenance burden for Cinder doesn't seem to have much payoff 16:20:41 winston-d: or... provide test libs on github if we so choose 16:20:49 what is wrong with having functional tests part of cinder that can be cherry-picked? 16:20:52 i worry about maintenance. who owns that? 16:21:10 kmartin: eharney that's what I was saying but guitarzan *chuckled* at me :) 16:21:15 thingee: agreed 16:21:20 so e.g. I decide to implement soem solidfire feature, then I have some test I can run 16:21:29 thingee: right. putting them in-tree makes people who aren't really that interested responsible.. 16:21:31 I don't see much maintenance burden from self-contained test cases 16:21:59 so as I started the conversation, I don't want to be responsible... 16:22:13 I don't know how the different back-ends are supposed to react/behave 16:22:15 jgriffith: One advantage of public test cases is they act as additional documentation. That isn't an arguement for having them in the main tree though 16:22:24 jdurgin1: every time we disable a driver test and we can't get hold of the maintainer has sucked 16:22:28 I do know how drivers are *expected* to behave however 16:22:53 DuncanT: again... umm... isn't that what the API is supposed to do? 16:23:01 DuncanT: and what tempest would flush out? 16:23:10 DuncanT: ensures that you adhere to the API correctly 16:23:19 jgriffith: but sometimes there are configuration tricks and stuff 16:23:26 winston-d: ok 16:23:26 jgriffith: Maybe just enhancing tempest would be enough 16:23:37 DuncanT: enhancing in what way? 16:23:38 +1 for tempest 16:23:51 I think I agree that tempest would be sufficient for the moment 16:23:53 jgriffith: There are lots of bits of code it doesn't cover 16:23:54 add functional tests to tempest? 16:24:06 you know anybody can contribute to tempest right? 16:24:10 jgriffith: But I guess just fixing that is good enough 16:24:15 Maybe it's just me, but what sort of functional tests are we talking about exactly? Tests that are using OS API? 16:24:22 jgriffith: Yeah, sorry, I was thinking out loud 16:24:29 DuncanT: no problem 16:25:04 matel: e.g. I want to test cinder backup api with Swift backend and then with Ceph backend. I need a functional test to do that. 16:25:06 matel: I think some folks are interested in things like performance testsing and more edge-case/vendor specific tests 16:25:32 matel: as well as the general API testsing that tempest should cover 16:26:07 If folks want to submit their functional tests to Cinder and there are people that want to review/maintain fine by me 16:26:14 but personally I really don't have time for that 16:26:31 but other people can certainly review/approve such patches 16:26:55 i see the value in functional tests. i just don't think we really care if a vendor specific solution works. 16:27:04 thingee: +1 16:27:46 jgriffith: I implemented a cinder driver a long ago, for XenAPINFS. So I defined a sort of library, that would be used by cinder, and I had functional tests for that library, meaning I had a running storage backend expected for those tests. Is this the kind of tests that we are talking about? 16:28:20 matel: I think that's part of it yes... there are multiple levels of detail here though 16:28:25 matel: yes, that is one use 16:28:45 Okay, because those tests does not really fit tempest I guess. 16:29:08 At least I put them to my own repo, copied the cinder stuff there, and ran the tests. 16:29:18 Ok, that's a 1/2 hour on this topic... perhaps we should move on? 16:29:23 sure 16:29:23 yes 16:29:24 Feel free to submit whatever you want 16:29:29 :-) 16:29:30 yeah, we have more to argue... 16:29:44 agreed 16:29:48 lets talk more about in the cinder room. 16:29:52 I suppose that sort of covered the "stress test for cinder backup" ? 16:29:52 ok next topic? 16:29:59 on a train atm on my phone :( 16:30:06 go for it 16:30:06 no not really 16:30:15 #topic stress test for backup 16:30:31 just wanted to tell you that we are goining to have a stress test in the next sprint (3 weeks) 16:30:31 mkoderer: have at it 16:30:39 sounds like functional test to me ;) 16:30:45 dosaboy: +1 16:30:48 so we reserved around 100 TB an a lot of machines 16:31:06 I already had a look to openstack-stress 16:31:17 but then I found out that we already have such a tool inhouse 16:31:36 oh good. in house tools. /me grumbles 16:31:37 so we will decide what we use or use both 16:31:47 mordred: :) 16:31:56 do you guys know about the stuff jog0 has done with FakeVirt in nova? 16:31:57 mordred: we will put it on github ofc 16:32:21 * mordred continues to grumble but will move on 16:32:30 mordred: yes, a bit (fakevirt) 16:32:32 but we will focus only on ceph 16:32:41 so no testing of lvm or something else 16:32:51 mordred: the debate is some folks would like to check in functional tests specific to their device 16:32:55 I think we will publish the results on our blog or somewhere 16:32:58 ah 16:33:09 ok so that all ;) 16:33:12 bbl 16:33:17 mordred: my stance is it should be tempest tests they can configure their backend and run 16:33:32 I would agree 16:33:32 mordred: or additional *special* tests on public github 16:33:39 mordred: but shouldn't go in the Cinder tree 16:33:43 mkoderer: do note that ceph backup is not expected to be performant until diff/incr backups are in place 16:33:50 is there an example of what type of 'special' test someone wants? 16:34:07 dosaboy: yes we know it 16:34:10 mordred: I've also mentioned in the past IIRC there's some things we can do if folks want to give you access to their lab/gear to do periodic tests? 16:34:19 mordred: so you are telling us that you have lots of h/w ready for stress tests and will publish your in-house tool as well. that's it? 16:34:21 but we already need to prepare the environment 16:34:41 sorry, it should go to mkoderer 16:34:43 jgriffith: yes. well, sort of - it's not that they give us access even 16:34:53 winston-d: no, I think we'er saying that vendor specific qual and testsing should be up to the vendor not OpenStack per-say 16:34:58 jgriffith: there is a mechanism whereby anyone can subscribe to the stream of events out of gerrit 16:35:05 jgriffith: and can run tests as they see fit 16:35:07 winston-d: yes that it 16:35:07 mordred: ahh... even better 16:35:11 jgriffith: and report the results back to the code review 16:35:18 jgriffith: like how smokestack works 16:35:25 it's a piece of cake to do 16:35:29 mordred: cool... smokestack was what I had in mind 16:35:37 mkoderer: do you have write the pref test case details down to some place? 16:35:48 there's even a jenkins plugin a person can use that will do all of the gerrit communication for them 16:35:54 So my point in all of this is that if they don't feel that tempest is sufficient they should submit enhancments to tempest 16:36:00 Yes sure 16:36:05 and use the method you described for their specific devices 16:36:08 yup! 16:36:31 http://ci.openstack.org/third_party.html 16:36:59 mordred: cool! 16:37:40 Ok... so that was pretty much the same topic after all :) 16:37:50 jgriffith: yes seems so ;) 16:37:51 so we have 38 minutes on functional tests :) 16:37:57 heh. but now that's two in 37 min.s so are rate is increasing 16:38:04 med_: haha! 16:38:14 hahaha 16:38:16 med_: always has a glass 1/n full 16:38:23 keep the flame going! 16:39:13 No more topic? 16:39:23 next topic? 16:39:41 #topic flatten volumes 16:39:46 winston-d: ^^ 16:39:54 yup 16:40:31 there is a bp intended to implement a public api extension to remove dependencies of a volume/snapshot 16:41:03 do we have a link? 16:41:05 here's the bp: https://blueprints.launchpad.net/cinder/+spec/add-flat-volume-api 16:41:19 winston-d: this seems need backend/storage support 16:41:38 certainly does 16:41:46 and avishay and I have different opinion about whether this kind of functionality should be public api or done internally 16:41:56 I think in the XenServer world, we call it coalescing 16:42:01 jgriffith: we skip topic refactor-backup-service? ;) 16:42:13 hrmm 16:42:17 mkoderer: I'll come back around to it 16:42:19 our backend won't support this 16:42:22 np 16:42:28 hemna: most won't 16:42:36 unless we clone the volume 16:42:45 Any backend can do it with a full copy... 16:42:51 should we be making public APIs for things that most backends won't support ? 16:42:51 so i would like to hear some feedback like do you guys want to see this implement as public api or not? 16:43:04 DuncanT: and any volume can continue to function as they do today by the same principal 16:43:04 hemna +1 16:43:16 but IMO, probably a lot of backend not support it natively 16:43:16 winston-d: netapp can support this 16:43:26 gpfs support it 16:43:28 winston-d: I've already spoken to this multiple times and my vote is -1 16:43:34 hemna: imo that's why this makes sense as an extension rather than a core api 16:43:36 jgriffith: noted 16:43:47 winston-d: :) 16:43:52 jdurgin1: any comment? i think this bp is from ceph use cases 16:43:55 I am in favor of it, but I'd prefer something that was less painful for those who can't support it 16:44:06 bswartz: which is? 16:44:11 I'm not sure 16:44:39 if this is the only way to support breaking teh dependency between snapshots and volumes then I want it 16:44:46 so this is meant to address an issue for some backends like ceph where a clone is dependent on the snapshot it came from 16:45:47 the dependency can be broken by effectively doing a full copy, but obviously having copy-on-write has its advantages as well 16:45:49 jdurgin1: wasn't that case already fixed with the rbd config option? 16:45:51 for us a clone is a complete copy and has no dependencies and in fact is the way to avoid dependencies from snapshotting volumes and creating volumes from snapshots. 16:46:06 jdurgin1: ahh, nevermind, handling both is harder :) 16:46:13 guitarzan: exactly 16:46:15 jdurgin1: can you remove dependency internally, transparent to end-user? 16:46:16 hemna: +1 as is true for SF and LVM 16:46:48 we can implement some code to allow particular backend support separation, but not easy if they not support it natively. 16:47:10 winston-d: no, that's the issue really 16:47:26 jdurgin1: why not? 16:47:35 winston-d: can't handle both cow and non-cow cases 16:47:51 so is the implementation of the ceph driver's clone creating this dependency because of the ceph apis and can that just be avoided by doing something else? Instead of backdooring this with creating a whole new public API that no one supports? 16:47:53 guitarzan: then why a public api can help? 16:48:12 hemna: no, this isn't the clone functionality 16:48:30 the BP says cloning 16:48:35 hemna: it's ceph's copy-on-write functionality when creating a volume from a snapshot 16:48:40 hemna: which is called cloning 16:48:46 gah 16:49:12 overloading of terms here then 16:49:25 to be clear, the dependency isn't really in Cinder then, it's in Ceph 16:49:30 is that accurate? 16:49:33 yes 16:49:37 so i still don't get why can't ceph do things like remove dependency when needed automatically 16:49:42 yes, and other drivers like nexenta have this issue too 16:49:48 winston-d: I think the question is now to tell when it's needed or not 16:50:08 s/now/how/ 16:50:11 winston-d: so like I mentioned on the review, a snapshot may have many volumes created from it 16:50:38 jdurgin1: ok, and? 16:50:50 winston-d: jdurgin1 and to be clear, creating a volume from snapshot in the other drivers doesn't not result in a dependent volume 16:50:50 winston-d: and we probably don't want to use up a lot more space and i/o doing full copies for all of them at once when someone wants to delete the snapshot 16:50:58 jgriffith: it does for nexenta 16:51:11 jdurgin1: sorry... Ceph and Nexenta 16:51:11 jgriffith: and maybe others, who knows 16:51:34 jdurgin1: my point as I've said before is that I think there could/should be a distinction between clone and snapshot 16:51:39 they're NOT the same thing 16:51:46 jgriffith: no, they're not 16:51:51 and that's exactly why we have this issue 16:51:52 jdurgin1: and when some user invoke such a call from public api, you are still in the same trouble? 16:52:18 jgriffith, +1 16:52:24 Hmm, I think the strive should be given to ceph and nexenta. 16:52:25 jdurgin1: i mean a user said 'flatten this snapshot' 16:52:29 jdurgin1: but for Nexenta and Ceph you're saying that clones are imlemented with internal snapshots 16:52:47 haomaiwang: what do you mean by "the strive"? 16:52:48 winston-d: no, the api would be 'flatten this volume' 16:53:05 ugh 16:53:08 winston-d: so that this particular volume does not depend on its parent snapshot 16:53:27 jdurgin, that's a clone :) 16:53:53 jdurgin1: cool. why would a user wants to do that do a volume? 16:54:21 jdurgin1: he can remove his created-from-snapshot volume anytime, right? 16:54:42 winston-d: yes, but he cannot remove the snapshot it was created from 16:55:11 winston-d: and flattening can also improve performance for some workloads 16:55:23 jdurgin1: right, so he should invoke the api saying 'flatten this snapshot' instead of 'flatten one of the clone' 16:55:51 winston-d: no, the snapshot is already flat 16:55:53 :) 16:55:58 well, only 5 min left. 16:56:11 we can continue this discussion in cinder channel. 16:56:31 we might need to stop digging into details during meetings 16:56:32 and we're back to what backends actually support flattening a snapshot ? 16:56:33 1 ? 16:57:25 let's continue in the cinder channel like winston suggested 16:57:26 full copy? 16:57:38 mkoderer: you are up for refactoring... 16:57:45 ok 16:57:52 #topic backup service refactor 16:57:53 3min left, sorry 16:57:59 so just quckly 16:58:03 about https://blueprints.launchpad.net/cinder/+spec/refactor-backup-service 16:58:17 thanks for the reviews 16:58:34 so we agreed to rename backup/service to backup/driver 16:58:43 +1 for that 16:58:49 I will work on the the next days 16:58:59 mkoderer: Only if you do the renaming magic so old config files still work 16:59:10 but we will keep the database field "backup.service"? 16:59:13 DuncanT: +1, should be trivial at this point 16:59:19 yes I know 16:59:28 this is what I am working on ;) 16:59:50 mkoderer: Database migrations are a pain in the arse on live systems, so if renaming the field is only to look pretty, don't bother 17:00:21 mkoderer: we shouldn't have anything driver/backend specific in the DB anyway 17:00:21 DuncanT: yes thats the point 17:00:34 timezup 17:00:57 med_: indeed 17:01:01 jgriffith: the service is stored there.. 17:01:04 mkoderer: don't change the db table :) 17:01:10 mkoderer: huh? 17:01:15 yes I won't ;) 17:01:20 mkoderer: my point is the driver info isn't, the service is 17:01:28 the service doesn't change 17:01:32 just the driver 17:01:40 ok 17:01:40 the whole point of abstraction is just that 17:01:51 mkoderer: no? 17:02:13 K... guess everyobyd is leaving 17:02:19 #endmeeting cinder