16:01:23 #startmeeting 16:01:24 Meeting started Wed Jul 18 16:01:23 2012 UTC. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:28 couple more from HP will be on in a couple of seconds 16:01:45 DuncanT: Cool, we'll start with last weeks actions to give them some time :) 16:01:55 #topic action items from last week 16:02:35 I had asked folks to take a look at devstack/exercises, anybody have a chance to do that? 16:02:55 :( 16:03:10 I think fergel verified that it fails for him as well... 16:03:25 Ok, I'll take it back up with the devstack team or see if I can find the problem/fix it myself. 16:03:41 Ahh, there's fergal now :) 16:04:07 the other item was quota implementation 16:04:24 I just moved forward with porting the quota code from Nova into Cinder 16:04:46 Meanwhile everett seems to have made some really good progress with the keystone implementation of quotas 16:05:01 We'll see if that gets picked up by Nova and follow suite 16:05:09 any objections? 16:05:26 cool 16:05:33 #topic progress updates 16:05:58 markmc and winstond have submitted a good chunk of work this week! 16:06:03 Thanks to both of them!!! 16:06:30 Most of it is related to using openstack.common and pulling out more dead code that came in from Nova 16:06:46 This is huge, it should really help clean up the code base A LOT 16:06:58 There's more to come here, but it's a nice big chunk 16:06:58 yeah exercise.sh failed twice for me on vm based installations 16:07:26 fergal: boot_from_volume.sh I'm expecting yes? 16:07:42 yup and there was the other one 16:07:58 http://192.168.0.154/ 16:08:04 boot_from_volume.sh did not work for me. 16:08:18 Vincent_Hou: ok... thanks 16:08:18 FAILED boot_from_volume 16:08:20 FAILED floating_ips 16:08:40 So there's definitely something different between jenkins/gate systems and what we're trying to run 16:09:17 I also pointed out that it fails on nova-volume as well as cinder, and I think it's a filter scheduling issue in the nova scheduler that's the root problem 16:09:39 We need to get that resolved this week... 16:10:04 Anybody have any other updates on things they're working on? 16:10:29 I have a blueprint up - https://blueprints.launchpad.net/nova/+spec/volume-usage-metering 16:10:51 jgriffith, we submitted a driver which you reviewed, but not sure that's considered an update. We'd be happy to help fix generic Cinder issues as well of course. 16:10:59 dricco: Assuming this is to tie back into ceilometer? 16:11:01 i will be working on making SM work in cinder for the rest of this week 16:11:15 avishay: yes, thanks! 16:11:18 I found a strange bug: https://bugs.launchpad.net/cinder/+bug/1023755 16:11:21 renuka: cool! 16:11:28 Launchpad bug 1023755 in cinder "Unable to delete the volume snapshot" [Undecided,New] 16:11:43 Sorry, what's SM? 16:11:54 Vincent_Hou: sadly it's not as strange as I would like :( 16:12:13 DuncanT: Xen stuff, you'll have to get the explanation from renuka :) 16:12:18 If I do I clean installation on a clean machine, this issue will be raised. 16:12:32 jgriffith: ceilometer can consume if it wishes 16:12:36 Vincent_Hou: I saw this during Essex 16:12:41 DuncanT: Storage manager volume driver 16:12:47 Vincent_Hou: I'm looking at this again today 16:12:48 but would like to get it into folsom and don't think ceilometer will be ready by then 16:12:54 renuka: Got you. Thanks. 16:13:11 Thx. 16:13:16 dricco: You may want to link up with them as that's something they've been asking for, at least if it's coordinated they may help you or at least be able to consume what you come up with 16:13:18 Thanks for the help in reproducing! :) 16:13:54 Vincent_Hou: That's interesting, the coorelation is the "first run" on a clean vm 16:14:10 yep 16:14:23 Vincent_Hou: Any luck trying to track it down? 16:14:25 Will do, I followed their design doc, and my design is very similar to how we get network stats at the moment 16:14:33 dricco: cool 16:14:34 i did try the update, it worked. 16:14:51 Vincent_Hou: by update what do you mean exactly? 16:15:12 jgriffith: misc bugs and I've pinpointed the problem to bad error reporting to the end user using the client. Talked to Durgin from ceph about rbd cloning. Will be providing an update to the driver soon 16:15:12 there is no error message, what I have got is the log of c-api. 16:15:20 jgriffith: hoping to get a first implementation in sometime next week 16:15:39 jgriffith: I haven't been able to get an error either. Going to debug it further today 16:16:04 thingee: sounds good, also I'm looking forward to the word from you and jdurgin 16:16:18 I mean update the existing code to the newest. 16:16:41 It is not the first time to run devstack. 16:16:47 I'm sure lots are. Cloning doesn't exist yet, but I'll at least have tests and the implementation done with cinder and glance 16:16:52 Vincent_Hou: So if it can't be reproduced in the latest code maybe we "accidentally" fixed it? 16:17:31 jgriffith: +1 16:17:32 No, please do a clean installsation on a clean machine with the latest code. 16:17:50 though I've had luck reproducing it in latest 16:17:55 "luck" 16:17:58 ;) 16:18:14 Ok, so bottom line is the bug is there and we need to figure it out, 16:18:23 To repro... fresh install and volumes > 1G 16:18:36 yeah was able to reproduce with 5G volume 16:18:38 Did anybody look to see if it repros using n-vol service as well? 16:18:53 yes 16:18:59 nova-volume too 16:19:17 Vincent_Hou: ok... I think this is the timing of the dd erase operation but not sure 16:19:37 If Vincent_Hou and thingee are going to work on it I'll leave it to them? 16:19:56 Vincent_Hou: thingee: Let me know if you need any help or find anything new 16:20:00 will do 16:20:03 is this with devstack installs? 16:20:07 i will stick to it 16:20:09 renuka: yes 16:20:12 yes 16:20:24 thingee: Vincent_Hou: Thanks!!! 16:20:32 doesn't devstack create a backing file of only 2GB, did you take care of that when you made a 5G volume? 16:20:46 renuka: default yes 16:20:47 I have tried both. 16:20:54 you can specify in localrc for larger 16:21:03 2G and more than 2G. 16:21:04 VOLUME_BACKING_FILE_SIZE 16:21:05 right, ok 16:21:09 yes 16:21:36 renuka: Also, it's not the create that fails but the delete so the backing file adjustments seem correct 16:22:16 jgriffith: I think renuka is saying you wouldn't be able to create a large enough volume to reproduce if the vg size is small. 16:22:21 I didn't have luck with 2G 16:22:36 You can try 1G 16:22:38 thingee: right... 16:22:57 this issue could be raised for 1G. 16:23:05 Vincent_Hou: that's what I tried to begin with and wasn't able to reproduce it. 16:23:20 you suggested in the bug to try a larger size 16:23:58 Well, please do a first install on a clean machine. 16:24:12 Ok so let's sync up later this week on that bug 16:24:22 It has been a new vm everytime :) 16:24:27 jgriffith: definitely 16:24:29 ok 16:24:42 any issue, please let me know. 16:24:42 Vincent_Hou: thanks your help with this! 16:24:47 sure thing 16:24:47 #action Vincent_Hou and thingee to work on bug https://bugs.launchpad.net/cinder/+bug/1023755 16:24:48 Launchpad bug 1023755 in cinder "Unable to delete the volume snapshot" [Undecided,New] 16:24:58 u r welcome. guys. 16:25:10 So something I need some help with if anybody has some cycles today :) 16:25:16 thingee: also quick question 16:25:43 I have the quota implementation, you can see it on my github: https://github.com/j-griffith/cinder.git and python-cinderclient.git 16:25:58 renuka: shoot 16:25:59 Sorry.. go ahead renuka 16:26:09 when you change the size of the backing file, assuming you are trying on kvm, you dont have to change anything else? cos on xen, i expect we would have to change the vpx disk size 16:26:49 renuka: But this is just a non-attached volume, so it shouldn't matter should it? 16:27:16 isnt it being carved out of the backing file on the devstack vpx (in xen I mean) 16:28:07 I am using kvm. I haven't had to, no. I just have that set on localrc and spin up the vm and run devstack 16:28:13 renuka: I don't think so... plus the default devstack on a vm isn't going to use xen anyway is it? 16:28:50 was just confirming (in the hope that someone out there is using xen) 16:28:54 ;) 16:29:09 renuka: If you want to help me set it up on my test system later today I'll use it :) 16:29:18 will do 16:29:36 renuka: cool! I have some "other" company business that I need a xen setup for 16:29:42 Ok... back to my quota code :) 16:29:46 renuka: I wouldn't mind a pointer to the lastest devstack/xen instructions 16:30:11 I've changed the default service type in python-cinderclient to volume.... 16:30:34 jgriffith: I can maybe have a look at it today but no guarantees 16:30:39 clayg: they are in a badly formatted state, in devstack/tools/xen/README 16:30:40 Before doing that I would read/write quota data from nova as it's the specified endpoint for compute 16:30:55 clayg: i promise to reply to the other email today... sorry about that 16:31:05 The problem now is I just try a simple quota-show and get http 404 errors 16:31:29 I've added the quota code as an extension in cinder/api/openstack/volume/contrib 16:31:39 But I'm missing *something* 16:31:49 renuka: thanks, np 16:32:07 If anybody is familiar with the plugin model or any of the api.openstack code please take a look and see if you can help me out :) 16:32:25 The other thing I'm trying to get turned on is cinder as default vol-service in devstack 16:32:25 jgriffith: can you give any hints what I'm looking for? plugin doesn't load? 16:32:42 jgriffith: I'm having a look, though I'm running out of day here so might not get any real time before morning 16:32:48 clayg: I don't know if that's the problem or not.... but it could be 16:33:15 clayg: I'm likely *missing* a step or config somewhere and I don't know where 16:33:38 jgriffith: you don't have a changeset in gerrit for quota's just your branch in your fork? 16:33:58 clayg: I have a draft in gerrit, and the latest code in my personal github 16:34:10 changes in both cinderlcient and cinder 16:34:19 jgriffith: sry, that sounded wrong - the best place to checkout the latest code is personal github or gerrit? 16:34:29 clayg: Ahh... personal github 16:34:40 jgriffith: ok, and cinderclient too - ok cool 16:34:47 You can try and run it in a devstack... just pull my cinder and cinderclient 16:34:52 clayg: Awesome!!! 16:35:06 Ok.. so the devstack, cinder/default issue... 16:35:32 I can't get this change in because for some reason the gate checks fail the volume tests just like we talked about earlier 16:35:46 What's odd however is that they're passing without my changes 16:36:06 I'll try to get with jeblair and dtroyer today on that.... 16:36:23 #topic queued reviews 16:36:40 Actually this looks good so not really an issue any more 16:37:02 If any of you submit code and it sits for a day make sure you ping me as I might have just missed it 16:37:11 #general status 16:37:23 #topic general status 16:37:32 Is there any status notifications for waiting reviews? 16:37:48 Or are you just polling the search? 16:38:04 DuncanT: Don't think so, I typically get the email then need to make sure I go in an poll the search 16:38:31 DuncanT: Not a problem but something we all should keep in mind is maybe once a day take a look in gerrit for reviews :) 16:38:37 when you post a review, you can also request a review by a person which will send an email 16:39:14 creiht: yes, good point, and it doesn't get lost in the hundreds of emails quite as easily 16:39:33 creiht: It's also nice because it shows up under the "my changes" section for anybody you added 16:39:38 yup 16:39:49 Ok, so general status... 16:40:18 Things are moving along ok this week, despite the disruptive email last week 16:40:35 creiht: Not referring to yours, yours was constructive and appreciated :) 16:40:39 lol 16:40:49 I think everybody knows which one I'm referring to ;) 16:41:24 Well a lot of this is a much broader question for openstack in general 16:41:32 It does bring up a good point, honestly I think in terms of compatability from the API's we've done the right thing and that's not something that should be a major issue 16:41:36 just bad timing for the cinder project 16:41:45 indeed 16:41:54 and I don't think anyone sees that as an issue 16:41:56 For folks with customized code it does make life a bit more... *difficult* 16:42:11 So the one thing that *is* a concern is the migration 16:42:35 Was there a consensus on what to do with cinder? Document, get feedback, and decide later? 16:42:36 Our top priority in the coming weeks is going to have to be a migration tool and test 16:43:04 avishay: I think we're in a state now where we have to prove we can do what we say before a decision is made 16:43:15 makes sense 16:43:18 avishay: In other words the migration and testing part 16:43:18 It isn't just migration in terms of a migration tool 16:43:43 DuncanT: ? 16:44:25 It is the fact that it needs addition bits (db, rabbit, load balancer) standing up, and doing that at the same time as a nova major version jumps multiplies risk 16:44:56 DuncanT: understand, but in terms of what the Cinder team has control over and can try to mitgate 16:45:00 We'd far, far, far rather migrate nova, then migrate to cinder 16:45:13 DuncanT: That's all I can do, the rest is up to the community 16:45:26 i.e. support both at production quality for a period 16:45:27 DuncanT: Yes, I understand and I think most folks with installations agree 16:45:29 jgriffith: has anyone suggested trying to freeze the volume tables schemea in nova & cinder until f-final to ease migration? (i don't see any migrations in cinder so far, so that's probably good?) 16:46:09 Freezing cinder while it is this early in settling could be limitting 16:46:12 clayg: Not exactly (re the freeze db suggestion) 16:46:32 Well, I think clayg is referring to the DB 16:46:58 Even freezing the db could hurt, e.g. the snapshot tables issues Fergal was looking at 16:47:17 DuncanT: I agree, that and implementing Chap which we really need to try and do before F release 16:47:18 DuncanT: I'm sure that it would be limiting, but as long as folks are going to throw around "it's a bit for bit copy" - I think they're pre-supposing it will remain mostly interoperable into the near future. 16:47:25 Freezing nova-volumes db and keeping an up-to-date migration tool seems reasonable 16:47:43 It isn't a bit-for-bit copy and hasn't been for a while 16:47:55 Ok, we could probabl devote an entire meeting to this topic.... 16:48:13 jgriffith: I think you are right in that you guys need a rock solid migration tool 16:48:16 jgriffith: We need beer to have a /proper/ meeting about it 16:48:19 To be honest there is enough work that needs to be done just to get equivalency with Nova still 16:48:22 but we also need a smooth window to migrate :) 16:48:29 DuncanT: I agree freezing nova-volume would work - I'm a little worried about this "back port all the fixes to nova-volume" plan :\ 16:48:36 * jgriffith likes the beer idea 16:48:44 * jgriffith getting one from the fridge now ;) 16:48:59 * creiht goes to buy my team bears ;) 16:49:09 * DuncanT sulks 16:49:10 hah... funny typo :) 16:49:25 did I mention we are hiring? :) 16:49:26 creiht: That's going to be then it's up to the community to decide how to proceed (IMHO) 16:49:34 jgriffith: agreed 16:49:45 creiht: I like bears too 16:49:53 creiht: But they're dangerous to have around when drinking 16:49:56 Not backporting anything except critical bugs to nova-volumes seems reasonable 16:50:12 DuncanT: That's my original proposal, but not my call 16:50:23 It's up to PPB and others 16:50:23 jgriffith: the best case scenario is that you guys do such an awesome job that everyone wants to move to it as soon as possible and nova-volume becomes an afterthought 16:50:24 DuncanT: +1 (double +1 for the ISCSIDriver) 16:50:33 that's what you should strive for :) 16:50:35 creiht: EXACTLY!!!!! 16:50:56 jgriffith: One approach is to keep a list of backports that haven't happened and invite somebody to step up to the plate if they care enough 16:51:20 DuncanT: The problem is people show they care in different ways :) 16:51:27 lol 16:51:29 hah 16:51:31 DuncanT: Some ways are more disruptive than others 16:52:06 We just want no regressions in nova-vol so we can stagger the upgrade... 16:52:15 New features being cinder only is fine 16:52:25 yeah I'm fine with new features only in cinder 16:52:34 but bug/security fixes should be back-ported 16:52:44 DuncanT: creiht: I don't disagree 16:52:56 'bug' is a broad category 16:53:00 Alrighty... moving along before we run out of time again 16:53:08 DuncanT: "critical bug" :) 16:53:08 Aye 16:53:16 #topic devstack 16:53:18 DuncanT: true, I'm in line with jgriffith's idea 16:53:36 I already talked about this, but there are a number of things we need to address 16:53:48 1. boot_from_volume.sh fails for n-vol and cinder 16:54:08 Well... really that's it but it causes issues in multiple places 16:54:15 ^ perfect example of who cares about fixing in n-vol - "expected behavior" 16:54:17 clayg: Error: "perfect" is not a valid command. 16:54:21 hah 16:54:47 clayg and virtbot are best buddies 16:54:55 the best of buddies! 16:55:33 What I don't understand is when did EVERYBODY stop running exercises.sh on their systems before submitting code :) 16:55:57 Anyway... 16:56:08 * jgriffith wines incesently 16:56:51 jgriffith: again, a broader general question for the openstack nova community :) 16:57:04 creiht: :) 16:57:10 creiht: I want discipline damn it!!! 16:57:13 hah 16:57:13 creiht: Just kidding 16:57:21 Ok... 16:57:25 #topic open discussion 16:57:30 Let'er rip! 16:57:41 jgriffith: wait... was there a 2.? 16:57:56 clayg: I don't count so well 16:57:59 heh 16:58:05 clayg: :) yes there was, but I changed my mind 16:58:32 clayg: Decided they were all the same (n-vol, cinder) and the issue with the gate job for the cinder change over 16:58:44 They're all the same failure 16:59:16 understood 16:59:28 Hey, nobody has anything to say? That's a first... 16:59:36 I have a question... :) 16:59:40 jgriffith: don't tempt me ;) 16:59:41 * jgriffith sneaks out while the getting is good 16:59:50 avishay: What's up? 17:00:01 please don't ask about docs :) 17:00:15 jgriffith: what's the deal with the docs? :P 17:00:20 For our driver, I understand that it won't go into nova-volume. What's the best way to distribute it and for OpenStack users to know that the system is supported? 17:00:28 * jgriffith punches clayg in the nose 17:00:38 Most will still use nova-volume for a while 17:00:40 * clayg rubs nose 17:00:51 avishay: Why won't it go into nova? You can surely modify it and submit it 17:00:59 avishay: In fact for now I would recommend it 17:01:05 I thought it's only bug fixes now? 17:01:23 avishay: what I would like and what happens are two different things :) 17:01:44 avishay: Nova team makes their own decisions/choices 17:01:50 Essex too? Or am I pushing my luck? :) 17:01:55 avishay: I'd recommend you hedge your bets 17:02:07 avishay: there I think you're pushing your luck :) 17:02:13 :) 17:02:19 backporting is a whole different story 17:02:30 anything else? 17:02:43 going once.... 17:02:43 OK will submit after cinder reviews are done - thanks! 17:02:50 avishay: No problem 17:03:17 avishay: The ones I have issue with are folks submitting nova-volume drivers and no cinder equiv 17:03:18 fwiw you really can maintain a driver outside of nova's source tree and distribute it yourself supporting and packaging for which ever versions of nova you'd like :\ 17:03:28 haha 17:03:37 clayg: Excellent point, I'm doing that now with an updated version of the SolidFire driver 17:03:40 nova's importutils don't care if the import path doesn't start with "nova" 17:04:21 jgriffith: don't forget they changed the import path for nova.log to nova.openstack.common.log :P 17:04:25 avishay: For our drivers it's even easier because really they just *drop_in* 17:04:43 jgriffith, yep, only imports are different 17:04:49 yeah I imagine our driver is very similar... mostly a pass trhough 17:04:52 through 17:04:53 clayg: yeah, you still have to maintain it and keep it updated :) 17:05:09 clayg, yea, that's what we'll do 17:05:10 You have to do the same work you do if you submit it 17:05:14 jgriffith: well we have to anyways if we want our service to remain working :) 17:05:36 creiht: you want things to work! Crazy talk! 17:05:40 jgriffith: actually more, because then you are locked into the nightmare that is contributing code to openstack :) 17:05:44 if it's in the tree you can make your tests fail - hopefully they don't just @skip :P 17:05:50 my main ask is that you map a migration based on what people (might) be doing from the docs at http://docs.openstack.org/essex/openstack-compute/admin/content/managing-volumes.html 17:05:55 creiht: Hey.. it's not that bad once you figure it out 17:06:14 I know that "everyone" has their own way of running volumes so a scripted way isn't a one-size-fits-all proposiion 17:06:15 n 17:06:26 I can't spell proposition :) 17:06:57 annegentle: ahhhh, that's going to be interesting 17:07:17 annegentle: Thanks for pointing it out... 17:07:46 #action jgriffith MUST at least start docs this week 17:08:02 #action jgriffith migration docs will also be needed, not just a script/tool 17:08:10 jgriffith: I went to the Austin OpenStack meet up and that was the discussion in the room 17:08:15 bye all 17:08:26 annegentle: makes perfect sense to me 17:08:36 yeah to me too 17:08:38 No to individuals are the same 17:08:43 two even 17:08:57 write a killer doc and your critics will have to simmer down :) 17:09:17 annegentle: They won't have to, but I don't have to argue with them, just give them a link :) 17:09:25 i think it helps. 17:09:34 cool... 17:09:40 Good meeting this week everyone! 17:09:55 Thanks for showing and and all the work you're all doing to help get Cinder going! 17:10:02 thx 17:10:18 dricco: I didn't skip your blueprint topic inentionally 17:10:26 dricco: Do you mind posting it again for next week? 17:10:35 Just ran out of time 17:10:56 dricco: Actually I'll put it on the agenda and make sure we talk about it more 17:11:10 #action add driccos blueprint to next weeks agenda 17:11:14 #endmeeting