16:02:06 #startmeeting cinder 16:02:07 Meeting started Wed Nov 27 16:02:06 2013 UTC and is due to finish in 60 minutes. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:02:11 The meeting name has been set to 'cinder' 16:02:12 Hey everyboyd! 16:02:14 o/ 16:02:15 everybody even 16:02:19 hi 16:02:20 hi 16:02:22 hi 16:02:26 Hello everyone 16:02:27 hello 16:02:30 hello 16:02:53 couple things on the agenda: https://wiki.openstack.org/wiki/CinderMeetings 16:03:19 anybody from Mirantis here to talk ACL? 16:03:38 jgriffith: that's under dec 4 ... not sure if on purpose or not 16:03:56 DOH 16:04:02 todays the 27'th 16:04:05 avishay: thanks :) 16:04:09 Ok then 16:04:10 :) 16:04:17 #topic update on volume mirroring design 16:04:40 OK, so I have a new etherpad up with a new design for volume mirroring 16:04:49 (it's linked from the original) 16:04:56 https://etherpad.openstack.org/p/icehouse-cinder-continuous-volume-replication-v2 16:05:00 Basic highlights: 16:05:16 1. it's controlled by volume types - not visible or controlled by non-admins 16:05:37 2. we're only dealing with 1 cinder deployment - not mirroring between multiple 16:06:11 I think it's pretty simple for a first pass, and would appreciate feedback on it before I implement too much 16:06:39 avishay: I like it personally 16:06:44 Mostly I'm unsure of which method is best for new database tables, which I'd like to hear feedback on 16:06:49 and I'd rather start with the simpler cases like this and build on them 16:07:07 jgriffith: +1 16:07:20 For example, should a second volume copy be a new DB table, or just use Volume? And should it be controlled by extra_specs or add a new table like QoS? 16:07:39 avishay: I'm also open to revisiting API commands for the user to select replication schemes later but I'd rather get this simple case in first 16:07:41 jgriffith: yup that's the plan. give the users something simple, and see what features are needed in the futures. 16:07:43 future* 16:07:47 at the summit it seemed like we were trying to solve stacks that exist on different planets. I'd rather not go down that rabbit hole 16:08:02 thingee: +1000 16:08:09 thingee: different planets sounds exciting, but yes :) 16:08:20 avishay: so backing up to the DB part 16:08:48 avishay: so if I'm reading this right -- the backend implements the mirroring and then informs cinder where the mirror copy is? 16:08:49 avishay: I was thinking it should be another instance of a volume object/table 16:09:19 jgriffith: i.e., another row in the Volume table? 16:09:27 avishay: correct 16:09:44 jgriffith: +1 16:09:47 bswartz: basically yes - cinder is not implementing the moving of the bits 16:09:56 avishay: and then either some additional columns that will be needed, OR an reference table specific for rep 16:10:16 jgriffith: i am leaning towards that direction, and we can take advantage of admin_metadata 16:10:32 avishay: which? The FK idea? 16:10:32 avishay: and more importantly, the decision of where to move the bits it out of cinder's control, correct? The driver decides and just tells cinder. 16:10:58 jgriffith: FK? 16:11:05 foreign key 16:11:33 jgriffith: I don't think FKs were planned. 16:11:36 bswartz: no, the scheduler will choose two backends - each driver will report who it can talk to 16:11:46 avishay:the information about mirrored entity is backend specific 16:11:46 jungleboyj: avishay what I'm saying is: 16:11:50 jgriffith: the idea of adding another row in the DB for the copy 16:11:57 there's likely going to be a need for some db info specific to rep 16:12:03 avishay: ahh, that's an important distinction 16:12:19 whether those become columns to the existing volume table or a fk in the volume table to a new rep table 16:12:26 jgriffith: yes, we can either add a new table for that or stuff it in admin_metadata 16:12:29 however I may be getting ahead of things 16:12:46 and I know thingee was working on some ideas for persistence without having to muck in the DB 16:12:49 not sure pros or cons 16:12:59 FK's are a PITA for migrations 16:13:29 not a big deal, just another moving part for DuncanT to deal with in his rolling upgrades 16:13:30 I don't want to get in a deep discussion about this - but is this consider an extension or not? I have ideas on allowing extensions to have tables and avoid adding more metadata...just curious 16:13:36 i'm thinking i should put up a patch for just the DB and get feedback on it before moving on 16:13:57 thingee: I believe this might be a core api function but not sure 16:14:09 thingee: the problem is it doesn't have to be implemented, which by def means NOT core 16:14:13 this isn't actually in the API 16:14:27 there is a new admin API for failover, but no user API 16:14:39 avishay: yeah, it's kinda of a "new" case 16:14:39 it's part of create_volume 16:14:48 avishay: part of the rest api or not, I think it'll be important to distinguish this now 16:15:09 thingee: to start with it'll be just another instantiation of volume-type 16:15:11 avishay: correct? 16:15:13 just with some ideas I have in how things can actually "plugin" to cinder. 16:15:26 that aren't part of the normal core of create, delete, detach, attach, etc 16:15:33 thingee: what do you mean by "another instantiation"? 16:15:42 avishay: that was me :) 16:15:48 yea sorry, mistyped :) 16:15:50 avishay: I meant... another use case for volume-types 16:15:56 * thingee is apparently louder than jgriffith 16:16:00 :) 16:16:27 thingee: i'd be very happy to hear about them 16:16:50 especially if it simplifies things :) 16:18:00 jgriffith: basically the admin should make a volume type with mirroring_enabling=True and mirroring_target_rpo=60, and that's it (or do something more like QoS - up to you all) 16:18:39 avishay: right, that's what I was getting at 16:18:42 * jgriffith don't talk so good 16:18:47 jgriffith: cool 16:19:03 thingee: what were your thoughts on plugging in? 16:19:10 avishay: and a category like we did with QoS might be a good idea from the start 16:19:15 * jgriffith hadn't thought of that 16:19:35 jgriffith: ok, i can do that 16:19:38 avishay: lets save it for when I have something properly written. I don't think it has to delay this. I just wanted to scratch the surface of what I think is core anyways and what plugins are 16:19:57 thingee: OK, looking forward to it :) 16:20:18 alright... that all sounds pretty good to me 16:20:20 anybody else? 16:20:29 avishay: anything else you want to bring up on the topic? 16:20:33 * thingee is the only one working at the office it seems so should be quiet to write something up 16:20:39 eventually i'd like to use taskflow for this as well, but i'll put up some initial versions and hope to get feedback early on 16:21:06 that's all i have, thanks! 16:21:12 avishay: thank you 16:21:23 avishay: so I briefly talked at the summit about my plugin ideas, and it heavily relies on taskflow in order to "inject" actions in core for manager, etc 16:21:27 #topic Mock RULES!! 16:21:42 thingee: sorry... I cut you off 16:21:43 thingee: yep i remember - sounds cool 16:21:55 jgriffith: you cut thingee off with his own topic :P 16:21:59 avishay: it would be great if new things coming in were ready with taskflow 16:22:05 I know :P 16:22:07 see how I am 16:22:09 +1 for moving to mock 16:22:09 * thingee is cutting himself off 16:22:18 dosaboy: +1 16:22:20 ha 16:22:32 thingee: yep - i plan on using taskflow before it's out of WIP status 16:22:45 so if you read the nova discussion, you'll see newer tests are going this direction. older tests, we'll get there 16:22:49 +1000 for mock from me... my experience has convinced me that mock works much better for most of our tests than the other methods 16:23:15 quick summary of why it's better for those of use who haven't used it? 16:23:26 avishay: +1 16:23:27 I personally spent a bit of time figuring out mox. docs weren't that great IMO. 16:23:29 though we already have +1002 for it :P 16:23:31 most of our stubs-based tests have a bunch of holes 16:23:32 py3 compat 16:23:36 it's in stdlib 16:23:48 it has awesome use of context managers 16:24:09 thingee: do you have a good starter link? 16:24:15 it's pretty much similar to mox, but different interface. none of this replaceall verifyall stuff 16:24:22 cool 16:24:22 p3 compat and the fact that mox is hardly maintined these days are the main factors for switching afaict 16:24:34 https://review.openstack.org/#/c/57843/3/cinder/tests/test_volume.py 16:24:42 dosaboy: +1 16:25:09 in that patch I use patching as a context manager and decorator 16:25:11 thingee: yep i saw that. was easy to read. 16:25:14 kinda neat I think 16:25:37 all the docs you need http://www.voidspace.org.uk/python/mock/ 16:25:51 for patching specifically http://www.voidspace.org.uk/python/mock/patch.html 16:26:41 thingee: thanks 16:27:10 sounds good... so we want to make this a req for all new tests 16:27:14 anybody disagree? 16:27:16 my proposal is to ask reviewers to start enforcing new tests in mock 16:27:28 thingee: +1 from me 16:27:43 * jgriffith knows he'll be the first to submit without doing this though :) 16:27:48 haha 16:27:53 i think we should speicifcally create tasks to convert to mock and share them among us 16:27:54 jgriffith: Have they officially made the same decision for Nova? 16:28:18 dosaboy: I'm down with that, but it doesn't fit in to my priority list at the moment 16:28:23 Just curious. 16:28:28 otherwise we end up with a hash and people will still be tempted to copy and paste the mox stuff 16:28:31 dosaboy: of course if that's something somebody would like to work on that would be great 16:28:39 dosaboy: understood 16:28:39 or code tests in a mox-like way 16:28:48 but yeah priorities ;) 16:28:58 jungleboyj: don't know.. and don't really care TBH :) 16:29:14 jgriffith: Fair enough. :-) 16:29:16 every project is try to do the same afaik 16:29:20 trying 16:29:20 jungleboyj: despite popular belief Nova does not rule the world 16:29:22 i'm sure converting all the tests would uncover a ton of bugs :) 16:29:43 avishay: our tests do a great job of testing the test code 16:29:52 dosaboy: Ok. I heard rumors but wasn't sure. 16:29:58 jungleboyj: http://lists.openstack.org/pipermail/openstack-dev/2013-November/018520.html 16:30:06 do we think it's actually possible to convert all of our mox uses to mock? (i.e. any missing capabilities?) 16:30:23 we may start -1 new test cases not using mock 16:30:37 start from there 16:30:52 can we mock people who don't use mock? (sorry, had to) 16:30:55 thingee: Thanks. That makes sense. 16:31:15 avishay: :) 16:31:18 LOL 16:31:36 Oh avishay ! 16:31:43 OK sounds good to me 16:32:39 alrighty... any other comments on this one? 16:32:49 dosaboy: as far as creating a task 16:32:53 to convert 16:32:55 eharney: I've spent time with both mox and mock, and they seem to have similar functionality. 16:33:00 I'm cool with that but time is a concern IMO 16:33:08 i work with the guy who wrote mock so can (attempt to) field issues if needs be 16:33:10 dosaboy: that's a LOT of work 16:33:15 thingee: yeah i have a question about your example test -- will ask after the meeting 16:33:48 dosaboy: do you want to do a bp? 16:33:56 i have not used it much myself yet though 16:34:13 or does anybody else have thoughts/opinions on reworking all the tests 16:34:18 also mox and mock comparisons here http://www.voidspace.org.uk/python/mock/compare.html 16:34:26 should be useful for those getting started 16:34:40 jgriffith: perhaps an etherpad to start so people can joit down ideas/concerns etc 16:35:20 dosaboy: https://etherpad.openstack.org/p/cinder-mock-conversion 16:35:22 thingee: https://etherpad.openstack.org/p/cinder-mock-conversion 16:35:30 should start by putting thingee 's links there 16:35:33 cool i'll keep an eye on it! 16:35:48 maybe everyone should clean his/her own table? start from driver owners, they should convert test cases to mock by the end of I ? 16:36:09 winston-1: maybe 16:36:23 winston-1: that's quite a tall order ;) 16:36:25 winston-1: -1 16:36:37 but when I look at our bug list I see other things I'd much rather we were working on TBH 16:36:44 jgriffith: +1 16:36:51 I'm all for using something new but changing existing stuff doesn't seem like a good idea 16:36:59 then by the end of J. ;) 16:37:01 jgriffith: +1 16:37:03 I say for now we just make new tests going forward convert 16:37:06 i think this should be like taskflow...use it for new stuff and gradually convert the old 16:37:07 see how things go 16:37:07 bswartz: +1 16:37:19 bswartz: it will be an issue for certifying python3 support though 16:37:21 winston-1: definitely by L :) 16:37:22 and when people are doing maintenance or updating an existing test they can feel free to update 16:37:43 avishay: personally, i'm fine with Z. :) 16:37:47 winston-1: haha 16:37:50 winston-1: LOL 16:38:16 if people would own certain modules and commit to doing a few test cases a week... 16:38:19 alright... let's start with what we have with hopes/plans to make a project wide conversion during I and J 16:38:24 it might go faster 16:38:25 Z is closer than you think 16:38:29 haha 16:38:44 2027 by my math 16:38:46 so ignore my stupid idea, let's focus on new testcases first 16:39:22 jgriffith: should I add to hacking our requirement on mock new tests *when* possible? 16:39:28 winston-1: not stupid...wish i had the time :) 16:39:38 i'd also covert stuff to taskflow if i had more cycles :) 16:39:46 avishay: i know, me too. 16:40:11 * thingee should do more since he doesn't have a kid yet 16:40:25 thingee: probably 16:40:25 i think we may encourage new contributors who are willing to do something to put their energy on test cases. 16:40:28 thingee: whoa... what? 16:40:32 thingee: you expecting? 16:40:39 no! 16:40:44 don't start rumors in the community 16:40:44 winston-1: that's a good point 16:40:49 thingee: too late 16:40:50 thingee: that means you should do less - go have fun while you can!! :P 16:40:57 oh the tabloids 16:41:02 thingee: just sent a congratulations email to lists@openstack.org 16:41:07 lol 16:41:22 :-) 16:41:26 thingee: and TMZ, Enquirer, MTV NBC, Fox news etc etc 16:41:28 thingee: reply to all already. 16:41:31 twitter 16:41:38 * jungleboyj is glad this meeting is on IRC so I can hear it over my yelling children. 16:41:46 LOL 16:41:50 haha 16:41:52 anyways... 16:42:00 alright... so we have a justification to move everything 16:42:07 just need to find the time/resources to do it 16:42:19 let's see how things go the next couple weeks and try to firm up a plan 16:42:19 ok /action thingee will update hacking - reviewers will be on the watch for new tests using mock instead 16:42:41 thingee: +1 16:42:49 I think doing a BP and pointing new comers to it as Low Hanging Fruit might be a good idea 16:42:57 jgriffith: +1 16:42:57 yep 16:43:05 great way to learn the code 16:43:06 Makes sense. 16:43:46 alrighty... 16:43:52 agree 16:44:02 #topic rate-limiting 16:44:13 so the comments there are interesting 16:44:29 I don't necessarily agree with everything Joe says... but he does make some valid points 16:44:34 yea this was just an FYI to DuncanT and whoever else was working on this .. i don't have much to add 16:45:12 * jgriffith has never been a fan of rate-limiting to begin with :) 16:45:20 but I do think it has valid uses 16:45:39 and if people don't care then they just don't use it AFAIC 16:46:04 It's an annoyingly hard problem to get right, and I agree the current implementation is of extremely limited use 16:46:13 agree 16:46:22 DuncanT: you live!!! 16:46:28 Just about 16:46:33 :) 16:46:44 alright, I'm not sure there's an action item here for us right now 16:46:53 hemnafk: had some interest here 16:46:58 and kmartin I believe as well 16:47:06 however they're setting this on the backend 16:47:50 we support rate limiting on the backend 16:47:59 no action item, just learn from nova's mistakes 16:48:24 this rate-limit is different from THE rate-limit I was doing in H, right 16:48:27 ? 16:48:44 winston-1: it's the hypervisor implementation of it 16:48:56 wait this is confusing 16:48:57 winston-1: yeah what John said 16:49:05 haha!!! 16:49:12 * jgriffith was completely confused 16:49:14 sorry 16:49:18 winston-1: you're correct 16:49:19 there is Data rate-limiting ( winston-1 in H ) , and API rate limiting ( DuncanT in I) 16:49:27 winston-1: this is NOT data rate-limiting 16:49:29 my fault 16:49:34 avishay: +1 16:49:36 wait, Nova remove api rate-limit 16:49:38 geesh 16:50:02 winston-1: it would appear so 16:50:14 phew 16:50:18 They removed naive rate limiting, which is basically what we said in the summit... the basic implementation isn't actually a lot of use 16:50:41 so there isn't much to discuss here, just pointed out the thread so we don't fall into the same holes as nova did 16:50:50 avishay: thx 16:50:57 and sorry everyone for my confusion there 16:51:09 Ok... 16:51:15 #topic metadata backup 16:51:20 dosaboy: 16:51:22 so just wanted to do a quick rfc on the metadata backup patch 16:51:29 since it has gone kinda quiet and I know it is a topic dear to everybody's heart ;) 16:51:36 LOL 16:51:37 it is targeted for I2 so just wanna make sure it does not stall 16:51:41 i reworked the implemetation to fit the last review 16:51:45 We're all behind on reviews I think 16:51:45 TSM guys are happy to implement it, I think nexenta too 16:51:52 so I've got Ceph covered and I think DuncanT said he would do Swift 16:52:15 dosaboy: My personal take is "retype, metadata backup and ACL" are priorities for us to look at this week 16:52:17 so this was really just an "anything else?" 16:52:23 I know ACL isn't *ready* per say 16:52:35 dosaboy: link plz 16:52:42 but I'd like to help Anastasia get that in sooner rather than later assuming everybody else still wants it 16:52:46 * jgriffith isn't sure 16:52:50 navneet_: https://review.openstack.org/#/c/51900/ 16:53:29 which is kind of a descent topic 16:53:35 #review priorities 16:53:41 #topic review priorities 16:53:59 Everybody have a look at dosaboy metadata backup patch :) 16:54:05 yay! 16:54:08 and avishay 's retype patch 16:54:20 woohoo 16:54:22 the retype loooks pretty good to me, I just need to sort through quota testing 16:54:23 and try to contain your excitement 16:54:47 i need to write some unit tests for it 16:54:50 anybody have things in flight they want to bring attention to and feel should be priority? 16:55:19 oh 1 more thing 16:55:20 dosaboy: not sure why your patch doesn't show up for me here: https://review.openstack.org/#/q/status:open+(project:openstack/cinder+OR+project:openstack/python-cinderclient),n,z 16:55:31 re the sync locking thing 16:55:34 also please check https://launchpad.net/cinder/+milestone/icehouse-1 16:55:43 sure 16:55:50 Note that 12/05 (next week) is our due date 16:55:53 Qin suggested we intoduce read/write locks as well as rthe EXCL we currenty have 16:56:01 i think its a good idea 16:56:09 but not surw how best to do it 16:56:19 should it be in oslo or direct in cinder? 16:56:44 Direct in cinder 16:56:57 dosaboy: direct and if Nova and others find it useful we can move it 16:57:02 DuncanT: that woulkd remove the serialisation issue 16:57:02 promote it 16:57:07 They can move to OSLO once they've been shown to be useful, well designed and that somebody else needs tehm 16:57:08 ok cool 16:57:18 DuncanT: +1 16:57:28 FWIW that was supposed to be how the whole OSLO thing worked 16:58:02 i'll create a new task for that so we can move 56442 along 16:58:12 Ok... any other comments here? 16:58:16 I have on other quick topic if not 16:58:24 shoot 16:58:34 #topic CI gate 16:58:44 So there's a new idea we want to try 16:58:56 no more "recheck/reverify no bug" allowed in the gates 16:59:04 any failure must have a bug associated with it 16:59:14 +1 (although I know it means more pain) 16:59:20 for both recheck and reverify? 16:59:21 dosaboy: it is 16:59:26 eharney: correct 16:59:36 ok 16:59:42 The reason being that there are too many things that get missed/untracked 16:59:53 and there's a common behavior of just "no bug" 16:59:54 was thinking this was only going to apply to reverify based on discussion i had seen 16:59:58 not necessarily cinder folks 17:00:00 but anyway 17:00:03 sounds good though 17:00:04 i've seen cases where there really is no bug (test script didn't start for some reason), but others where people say "no bug" and a unit test failed 17:00:13 jgriffith: or have a general bug number for no bug? 17:00:22 avishay: in those cases we should file a bug against CI 17:00:31 jgriffith: OK will do 17:00:32 avishay: I ran in to that one the other day and that's what I did 17:00:50 I've also started tagging bugs with gate-failure 17:00:57 for those who don't have it - http://status.openstack.org/rechecks/ 17:00:58 when appropriate to help sort things a big 17:01:18 eharney: just doing reverify only frankly misses a good chunk of data IMO 17:01:36 I don't think it's that difficult to put forth a little extra effort here 17:01:44 i will admit that it isn't always obvious whether to file a bug against ci, tempest, the project, etc. if we're going to start filing more strange looking failure bugs, is there a recommended place to start when we don't know which is relevant? 17:01:45 and the data collection is very helpful 17:01:51 that's time 17:02:03 eharney: when in doubt tempest with the test name 17:02:08 k 17:02:10 testname, and link to logs 17:02:11 ok 17:02:13 thanks everyone 17:02:16 #endmeeting