20:59:35 #startmeeting reddwarf 20:59:36 Meeting started Tue Apr 2 20:59:35 2013 UTC. The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:59:37 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:59:39 The meeting name has been set to 'reddwarf' 20:59:46 hello 20:59:52 hello 20:59:55 as usual, >>> time.sleep(120) 21:00:12 hi 21:00:19 hola 21:00:27 hey there 21:00:49 hi 21:01:03 hello 21:01:04 present 21:01:22 #link https://wiki.openstack.org/wiki/Meetings/RedDwarfMeeting 21:01:30 #link http://eavesdrop.openstack.org/meetings/reddwarf/2013/reddwarf.2013-03-26-20.59.html 21:01:47 greetings 21:01:55 greets 21:02:07 we ready to get this party started? 21:02:14 yup 21:02:25 #topic Action items 21:02:32 * juice is doing a shot 21:02:36 nice 21:02:41 someone snag grapex 21:02:44 where's the alcohol? 21:02:45 Let's do it. 21:02:50 he's trying to resolve his connection 21:02:51 and smack him w/ a trout 21:03:01 ok datsun180b, still smack him w a trout, hes on a mac 21:03:10 so ill skip him for now 21:03:24 my action item is next, and i actually added it to the agenda today 21:03:33 so ill skip it till then (action / action items) 21:03:54 vipul: yer next, patch for backUps to database-api 21:04:09 yea, haven't gotten around to it 21:04:16 promise to do it this week! 21:04:28 #action Vipul to publish backup API to database-api 21:04:28 cool, can u re-action it 21:04:42 so backupRef vs backupUUID 21:04:58 i believe we decided backupRef was fine right, but it could jsut be a uuid no biggie? 21:05:01 oh nm 21:05:04 i emailed jorge 21:05:06 he never emailed me back 21:05:08 he hates me 21:05:13 #action SlickNik to finish publishing security groups API to database-api 21:05:13 ill send him another email 21:05:19 <3 SlickNik 21:05:35 I remember I started on that one, but I still have a couple of changes to it. 21:05:40 #action get ahold of jOrGeW to make sure about backupRef vs backupUUID 21:05:43 SlickNik yea i think that was abandoned 21:05:57 everuy openstack project seems to do this shit differently 21:06:02 in terms of uuid vs ref 21:06:06 I have the change, just haven't gotten around to making the couple of changes I wanted. 21:06:08 i know vipul... 21:06:11 there is no standard 21:06:14 its terrible 21:06:23 yeah, sucks. 21:06:32 no wonder George Reese is always putting the smack down 21:06:34 personally, i think of a ref being a remote system 21:06:39 oh god nooooooo 21:06:45 he's on my twitter feed 21:06:49 big time complaints :) 21:06:54 Who's George Reese? 21:07:04 enstratus 21:07:04 SlickNik: search your openstack mail 21:07:08 And why are his peanut butter cups awful? 21:07:17 grapex: !!! 21:07:24 hub_cap: That was awesome 21:07:25 hub_cap: will do… 21:07:33 What's up? 21:07:35 ok so back to ref vs uuid, in my brain 21:07:38 a ref is remote 21:07:41 and a uuid is local 21:08:03 lacking context, it seems the 'uu' part of 'uuid' disagrees with that 21:08:15 nice 21:08:17 heh... had to bring that up 21:08:32 sorry my definition of local is not correct 21:08:39 local being controlled by reddwarf 21:08:47 as opposed to controlled by a 3rd party system 21:08:53 yes uu still applies :) 21:09:02 in that case we should go with UUID then 21:09:12 well thats what im leaning toward 21:09:16 but let me just touch back w/ jorge 21:09:25 its fairly easy to change right? 21:09:30 Yea pretty minor 21:09:31 how abut backupID? 21:09:32 it's ref now 21:09:38 not UUID 21:09:40 Id might be better than uuid 21:09:42 yea that's probably better 21:09:47 It's ref now, but easily changed... 21:09:59 so BackUpId 21:10:00 :P 21:10:00 I like backupId 21:10:06 Sorry gang, my IRC client had a visual glitch- are we talking about using ID or UUID for what's now known in the client as "backupRef?" 21:10:09 k, we'll wait for Jorje? 21:10:11 jorge 21:10:20 ya but lets lean heavily toward backupId 21:10:21 yes, grapex 21:10:26 Ok 21:10:30 steveleon: can we change that now then? 21:10:35 esmute ^ 21:10:38 grapex: lets go back to yoru action item 21:10:50 xml lint integration in reddwarf grapex 21:11:22 Sorry, still nothing so far. Had a conference last week which threw me off track. 21:11:30 fo sure, plz re-action it 21:12:04 Ok 21:12:10 ok so lets say we are done w action items 21:12:22 yeah we can change it.. 21:12:33 i will have to change the rd-client that just got merged a few hours ago too 21:12:34 #topic Status of CI/jenkins/int-tests 21:12:43 esmute: okey 21:12:48 esmute: thanks 21:13:08 So not having a int-test gate has been killing us it seems 21:13:11 vipul: its all yo mang 21:13:28 SlickNik and I are working on a jenkisn here at HP that will listen to gerrit triggers 21:13:33 and give a +1 / -1 vote 21:13:41 ok it can be nonvoting too 21:13:47 got it sort of working, but the plugin we use to spin up a VM need some love 21:13:49 if thats hard ro anything 21:13:53 AHH 21:13:54 should it be more like check/cross then? 21:13:58 are u using the jclouds one? 21:14:06 no, home grown 21:14:06 datsun180b: j/y 21:14:09 jruby thingie 21:14:11 considering that builds that don't pass int-tests aren't worth shipping? 21:14:12 oh fun vipul 21:14:16 nope, it's something that one of the folks here came up with. 21:14:19 datsun180b: correct 21:14:29 Yea but i think we want voting 21:14:34 SlickNik: do you guys adhere to the openstack api internally? if so the jclouds one is bomb 21:14:55 hub_cap: We shoudl try the jclouds one.. honestly haven't event tried it 21:15:00 Yeah, it needs a couple of changes to be able to pass the gerrit id's from the trigger to the new instance it spins up. 21:15:01 its great 21:15:10 itll spawn a server, if it fails itll spawn a new one 21:15:13 it sets up keys to ssh 21:15:21 it does a lot of work for u 21:15:23 hub_cap, do you have a link to the jclouds plugin you speak of? 21:15:24 one other thing missing is checking to see if tests passed or not.. 21:15:44 https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin 21:15:46 #link https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin 21:15:47 currently can run them, but no check to see if it worked properly 21:15:53 hub_cap: thanks! 21:16:02 grep for OK (Skipped=) 21:16:05 vipul: ahh, i think that the jclouds plugin will fix that 21:16:07 at minimum 21:16:16 yea, that's something we're trying to get added to our jruby plugin 21:16:23 itll fail if the int-tests emit a error code 21:16:24 hub_cap: Jclouds does that already? 21:16:34 even better 21:16:37 well u just tell jclouds to exectue X on a remote system 21:16:40 and if X fails, it fails teh job 21:17:18 hub_cap: so jenkins plugin is building ajenkins slave? or arbitrary vm 21:17:28 cuz i don't care for the jenkins slave.. just want a vm 21:17:56 vipul: thre is not much difference between them, but it can easily do arbitrary vm 21:18:14 vipul: Does StackForge CI use the jclouds plugin and make it an official Jenkins slave or does it just create a VM without the jenkins agent? 21:18:16 i like the idea of int-tests running on a machine that doesn't persist between builds and so doesn't rely on manual monkeying for tests to work 21:18:25 they have a pool of servers grapex 21:18:30 it _is_ a slave in terms of jenkins but thats convenient for making sure the node comes online etc 21:18:31 not sure exactly how they allocate them 21:18:36 datsun180b: mee too. 21:18:52 They have home-grown scripts to allocate them… 21:18:55 datsun180b: yep, fresh instance each time 21:19:14 anyhoo, i say look into it 21:19:16 SlickNik: home-grown instead of using the jenkins agent? 21:19:20 it may or may not 21:19:23 work for u 21:19:26 I'm not pro or con Jenkins agent btw, just curious 21:19:35 grapex: the ci team? ya its all homegrown :) 21:19:43 Yea... so still a WIP.. i think we need to give this a bit more time.. 21:19:50 BUT we're getting close 21:19:56 last week all tests passed 21:20:03 hell yes 21:20:15 even if its nonvoting and it triggers and we can just look @ it b4 approving 21:20:18 thats a good step 1 21:20:24 lets just get it runnin 21:20:33 We get the voting part from the gerrit trigger. 21:20:36 so we can stop comitting code that fails 21:20:43 yep, can't wait 21:20:44 +40 21:20:57 +! 21:20:57 im fine w/ it always voting +1 since it doesnt tell if it passes or fails yet 21:21:00 And I've set up the accounts to be able to connect to gerrit. 21:21:05 lets just get a link put up 21:21:26 OH and need to do openID integration 21:21:27 yeah, we probably need to run the int-tests locally before checking 21:21:32 rather than taking it to the finish line fully working 21:21:41 #action Vipul and SlickNik to update on status of VM Gate 21:21:43 I meant checking in 21:21:45 lets get a baton pass by getting it running for each iteration asap ;) 21:21:46 agreed hub_cap 21:21:59 We need this goodness! :) 21:22:02 yup 21:22:17 #action stop eating skittles jelly beans, they are making me sick 21:22:26 ok we good on ci? 21:22:27 thanks for actioning, Vipul 21:22:32 i think so 21:22:35 #Backups Discussion 21:22:38 status first 21:23:00 i thnk juice / robertmeyers / SlickNik you're up 21:23:29 backups are good, lets do it 21:23:34 So we got a sweet merge from robertmyers with his streaming/mysqldump implementation... 21:23:56 to our shared work in progress repo. 21:24:04 robertmyers: lol 21:24:09 we need a good way to run the restore 21:24:40 are we trying to get the backup _and_ restore in one fell swoop? 21:24:46 or are we going to break it up to 2 features? 21:24:49 I am working on hooking up our innobackupex restore implementation to it. (Testing it out now, really) 21:24:55 since we havent nailed the api to omuch for the restore etc 21:25:03 i think we have hub_cap 21:25:06 that's the backupRef peice 21:25:08 well, we should at least have a plan 21:25:14 I think esmute has the API/model pieces for both ready to go. 21:25:15 i agree we need a plan 21:25:22 oh well then SlickNik if its taht easy 21:25:41 I'll let esmute comment. 21:25:42 :) 21:25:43 hub_cap: so the backupRef vs backupId discussion is related to restore 21:25:53 yup... just need to do some renaming from ref to id 21:25:54 where we create a new instance, and porivde the backup ID 21:25:58 that's the API 21:25:58 there may be extra things we need like to reset the password 21:26:01 But the plan was to check with him and push those pieces up for gerrit review. 21:26:02 am i missing something? 21:26:27 robertmyers: could they do that after the restore? 21:26:30 robertmyers: I was wondering about that. 21:26:56 we could do it automatically after the restore 21:27:15 which password? the root user? os_admin? 21:27:15 should they get the original password by default? 21:27:17 they may want to use all the same uses/etc 21:27:29 robertmyers: who would the user get the new password 21:27:54 I think the root mysql user password will need to be reset 21:27:56 juice: it comes back in the POST response body for create 21:28:06 so i thought we'd slide the restore piece into the current 'prepare' workflow which i beleive does that after the data files ar ein place? 21:28:10 robertmyers: yes it would 21:28:19 or you can do a separate call as robertmyers said 21:28:34 and it shouldnt be enabled by default since new instances dont come w/ root enabled by default 21:28:52 got it 21:28:57 <3 21:29:18 and the osadmin user/pass might be goofy 21:29:46 im assuming we are pullin in the user table 21:30:03 so given that, we will have a user/pass defined for that, as well as a root pass 21:30:09 that is the plan, a full db backup 21:30:12 Yeah, we're pullin in the user table as part of restore. 21:30:25 so we might have to start in safe mode, change the passes, and then restart in regular mode after writing osadmin to the config 21:31:03 What's the behavior if a db with root enabled is backed up? 21:31:07 (on restore) 21:31:23 id say do _not_ enable root 21:31:26 no matter what 21:31:34 #agreed 21:31:37 cuz that needs to be recorded 21:31:44 and it becoems a grey area for support 21:31:58 So the restored instance is the same except with root _not_ enabled…? 21:32:16 correct 21:32:24 since enable root says "im giving up support" 21:32:31 and a different password for os_admin 21:33:02 gotcha vipul. 21:33:34 so great work on status team 21:33:38 now division of labor 21:33:41 whos doin what 21:33:45 cuz i aint doin JACK 21:33:47 Just one clarification. 21:33:50 wrt backups 21:33:52 yes SlickNik sup 21:34:06 So it's fine for them to backup from an instance on which they have given up support and restore to an instance for which they will have support. 21:34:41 hmm thats a good point of clarification 21:34:43 that's tricky.. 21:34:50 cuz we don't know what they may have changed 21:34:56 ok so given that 21:35:04 lets re-record that root was enabled on it 21:35:12 and keep the same root pass 21:35:20 sounds like the right thing to do 21:35:22 really its the same database 21:35:30 imsplitbit: for some backup strategies, doesn't the user info get skipped? 21:35:33 so the ony pass that should change is os_admin cuz thats ours 21:36:06 Maybe I'm thinking back to an earlier conversation, but I remember this came up and the idea was the admin tables wouldn't be touched on restore. 21:36:21 grapex: i think that was b4 mysqldump righ? 21:36:46 grapex: you can choose which tables to include in backup 21:37:00 hub_cap: Maybe, it seems like I might be speaking crazy here. 21:37:09 I think we want to do all and get the users 21:37:09 i think grapex is alluding to a conversation that was had regarding a internal product here 21:37:11 vipul: For iter 1 let's just do the full thing and remember what the root enabled setting was 21:37:41 yup just make sure to re-record the root enalbed setting w/ the new uuid, and leave teh root pass the same 21:37:44 hub_cap: No, it was earlier than that, talking about import / export... n/m. For iter 1 you guys are right and we should record the setting. 21:37:45 grapex / vipul / hub_cap: I like grapex's iter 1 idea for now. 21:37:46 update os_admin 21:37:46 not sure about mysqldump.. but xtrabackup supports expressions to exclude /incdlue tables 21:37:57 vipul: same w/ mysqldump 21:38:05 so then we could make a call now 21:38:15 if we want to start with a fresh set of users each time 21:38:19 then we just exclude it now 21:38:32 naw i dont think so, it doesnt make sense to _not_ include users 21:38:39 For imports it might- 21:38:49 even if so, u still have the root enalbed issue 21:38:51 that's down the road though 21:38:51 well, I think that can be set by the implementation 21:38:51 and to get manageability on restore back 21:38:54 Well, from a users perspective, if I've done a bunch of work setting up new users, I don't want to have to redo that on restore, though... 21:38:58 yea nvm 21:39:02 you still have other tables 21:39:09 lets go w/ all or nothing as of now :) 21:39:16 we are devs, we dont know any better :D 21:39:24 let some mysql admins tell us we are doing it wrong later 21:39:25 ;) 21:39:42 * grapex blows horn to summon imsplitbit 21:39:42 we can add a flag later to support this use case 21:39:49 well, right now the command is plugable 21:39:51 #agreed w/ grapex iter 1 21:39:57 i say we move on 21:39:59 so it can be easily changed 21:40:06 tru robertmyers 21:40:08 division of labor 21:40:11 sounds good, thanks for clarification. 21:40:11 whos doin what 21:40:35 * robertmyers will work on all the fun parts 21:40:39 lol 21:40:46 haha 21:40:58 I'm working on the innobackupex pluggable part. 21:41:40 right now I'm looking at the backup model to see if we can remove logic from the views 21:42:00 like a test to see if a job is running 21:42:25 anyone doing the streaming download? 21:42:30 robertmyers: what is your intention for work on the restore? 21:42:50 vipul: I think the download is a lot more straightforward than the upload 21:43:06 well, I was thinking that we create a registry, and look up the restore process from the backup type 21:43:25 since swift handles the reassembly of the pieces … or at least that is what I read in the documentation 21:44:07 robertmyers: do we do that or just mirror the configuration that is done for the backup runner? 21:44:18 juice: yes if you download the manifest it pulls down all 21:44:36 id say lets think we are doing 1x backup and restore type for now 21:44:36 robertmyers, do we need a registry? It makes sense to have the restore type coupled with the backup_type, right? I don't see a case where I'd backup using one type, but restore using another... 21:44:45 correct, for now 21:44:51 in the future we will possibly have that 21:44:57 well, since we are storing the type, one might change the setting over time 21:44:57 hub_cap, that was my thinking for now, at least. 21:44:59 a user uploads a "backup" of their own db 21:45:29 robertmyers: i dont think that we need that _now_ tho 21:45:42 dont' we already have the use case of 2 types? 21:45:43 that could happen in teh future, and we will code that when we thik about it happening 21:45:47 xtrabackup and mysqldump 21:45:51 No i'm talking about us changing the default 21:45:52 I agree 21:46:11 Let's just put in types now. 21:46:11 vipul: but xtrabackup will have its own restore, and so will mysqldump right? 21:46:21 grapex: types are in teh db i think arelady 21:46:22 right? 21:46:27 right but you need ot be able to look it up since it's stored in the DB entry 21:46:27 so we store the backup_type in the db and us that to find the restore method 21:46:42 so what wil this _restore method_ be 21:46:46 it's really a mapping.. 'xtrabackup' -> 'XtraBackupRestorer' 21:46:48 grab a file from swift and stream it in? 21:46:59 vipul: yes 21:47:02 the xtrabackup file cant be streamed back in? 21:47:06 like a mysqldump file 21:47:17 well, that part will be the same 21:47:29 it needs to be streamed to xbstream to decompress it. 21:47:30 this discussion is do we use configuration to more or less statically choose the restore type or do we use some component that chooses it based off of the backup type? 21:47:33 but the command to run will be differrent 21:47:39 But then it has an extra step of running the "prepare" 21:47:49 im confused 21:48:03 w/ xtra do u not do, 1) start reading from swift, 2) start streaming to mysql 21:48:06 download then backup 21:48:09 like u do w/ mysqldump 21:48:13 download is the same for either case 21:48:22 mysql < dumpfile 21:48:26 hub_cap: no... you don't pipe it in 21:48:32 backup process may vary yes? 21:48:34 thats just terrible 21:48:38 :) 21:48:42 you have to 'prepare' which is a xtrabackup format -> data files 21:48:53 then you start mysql 21:49:22 hub_cap: db consistency isn't guaranteed unless you run prepare for xtrabackup. 21:49:26 ok lets go w/ what robertmyers said then.... i thoguht they were tehe same 21:49:26 same download + different restore + same startup 21:49:36 is it the same startup? 21:49:39 one assumes mysqsl is up and running 21:49:47 other assumes it's down.. and started after restore 21:50:23 Seems like it's different. I think for mysqldump, mysql already needs to be running so it can process the logical dump. 21:50:27 ok so lets not worry bout the details now that we _know_ its different 21:50:29 vipul: yeah. 21:50:44 lets just assume that we need to know how to restore 2 different types, and let robertmyers and co handle it accordingly 21:51:08 robertmyers: so where do store the dump? 21:51:18 assume that there is enough space in /vda? 21:51:31 or actually you stream directly to mysql 21:51:46 where xtrabackup streams directly to /var/lib/mysql 21:51:47 good question, we may need to check for enough space 21:52:11 we can see if streaming is possible 21:52:12 mysql < swift download 'xx'? 21:52:19 mysqldump shouldnt store the dump i think 21:52:25 stream FTW 21:52:31 I think you may be able stream it to mysql directly. 21:53:26 lets assume yes for now, and if not, solve it 21:53:38 i _know_ we can for mysqldump 21:54:12 moving on? 21:54:27 oh, one other clarification. 21:54:31 kk 21:55:06 I'll probably be done looking at the xtrabackup backup piece by today/tom. 21:55:24 and so juice and I can start looking at restore. 21:55:36 cool 21:55:41 so on to notifications 21:55:43 #topic Notifications Plan 21:55:52 vipul: batter up! 21:55:52 cool, thanks. 21:56:14 I updated the wiki with our notifications 21:56:17 SO thanks to robertmyers for filling out hte info in wiki 21:56:25 i wanted to see where this is on your radar 21:56:26