21:05:04 #startmeeting reddwarf 21:05:05 Meeting started Tue May 7 21:05:04 2013 UTC. The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:05:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:05:08 The meeting name has been set to 'reddwarf' 21:05:11 lets put it on the agenda 21:05:17 sure 21:05:19 #link https://wiki.openstack.org/wiki/Meetings/RedDwarfMeeting 21:05:22 someone edit plz 21:05:33 I'm on it 21:05:34 got it 21:05:47 #link http://eavesdrop.openstack.org/meetings/reddwarf/2013/reddwarf.2013-04-30-21.03.html 21:05:51 so lets start with 21:05:57 perfect timing 21:05:59 #topic update to action items 21:06:12 wow there are virtually non 21:06:13 e 21:06:29 the only real tangible one is SlickNik 21:06:30 we didn't record many last time 21:06:42 did u reset the exectors on jenkins 21:06:46 yes 21:07:07 We now have 4 executors, and they seem to be doing well. 21:07:11 So we're good with that. 21:07:38 sweet 21:07:47 ok well done w/ Action items 21:07:50 HA 21:07:56 i've still seen some random failures but running again seems to be fine 21:08:06 #topic TC 21:08:12 well we are incubated 21:08:17 WOO 21:08:18 +1 21:08:20 good job all! 21:08:48 nice work hub_cap fielding 21:08:58 so what did we have to agree to? 21:09:06 a lifetime of solitude 21:09:08 answering emails 21:09:20 and fighting about flaygs 21:09:25 figured 21:09:34 :P 21:09:43 #topic OpenVZ 21:09:47 imsplitbit: wanna talk about the ovz status? 21:09:57 No listening to the sound of nails on chalkboard, at least... 21:10:31 hah 21:11:07 imsplitbit: is on his way 21:11:07 ovz status 21:11:09 ... 21:11:14 nice 21:11:16 oh the packages? 21:11:17 its in braille? 21:11:22 S? 21:11:31 morse code :) 21:11:50 theres a ppa up, I'm still trying to figure out how to get it to build for multiple archs and distros 21:12:01 No, "..." is JRPG for "Saying nothing" 21:12:34 jrpg? 21:12:45 video games 21:12:47 anyone with extensive debian packaging background is more than welcome to send me a chat and help cause this is my first time 21:12:48 ha SlickNik 21:12:49 lol 21:12:52 Japanese role playing game. 21:13:16 sorry to distract 21:13:17 ugh didn't know there were categories of rpg's 21:13:20 imsplitbit: ... 21:13:24 ppa is here 21:13:28 link it 21:13:29 #link https://launchpad.net/~imsplitbit/+archive/openvz-nova-driver 21:13:35 woot 21:13:36 I was dang... 21:14:02 but yeah, if anyone has packaging exp please hit me up tomorrow 21:14:05 ok nice work. cant wait to integrate it into our workflow @rax 21:14:05 19 seconds ago nice 21:14:09 I'd love some expertise on this 21:14:10 imsplitbit: do u think i can help? 21:14:19 i can try at least 21:14:26 sure we can look at it 21:14:36 we can also ask zigo, who has mad debian packaging skills 21:14:38 more than one set of eyes is always good 21:14:44 kk 21:14:47 ill use my 3rd eye 21:14:55 I don't even want to know 21:14:55 #topic Jenkins woes 21:14:57 heh 21:15:03 SlickNik: brief update plz sir 21:15:19 Most of woes should be fixed by now. 21:15:44 maybe not jenkins related, but anyone looking into the int-tests? 21:15:56 concerned that nearly every patch is failing 21:15:59 This includes the issue with the stderr logging in the plugin and the reg-exp for success. 21:16:08 i tried to run them last night and it failed like 10+ tests 21:16:17 SlickNik: nice on the regex 21:16:18 THere is still issues with some tests failing on-and-off. 21:16:40 So if we were to move to openstack ci tomorrow.. we still wouldn't be much better off without good tests 21:16:40 Before I left yesterday the only failure example I could see was, from what I can tell, in the client 21:16:41 ^ I got a failure on the resize tests 21:16:42 The two primary ones being "Restart Mysql" 21:16:53 vipul: #agreed 21:17:01 and "Resize down" 21:17:08 we need to make sure those are stupid solid before moving to official tests 21:17:10 datsun180b: that one was because of a rebase issue 21:17:34 datsun180b: vipul and I figured out that one last evening. 21:17:34 vipul: oh good, i was worried i couldn't see more of what was running to diagnose further 21:18:35 So we really need to either: 1. solidify those 2 tests, or 2. take them out / replace them with equivalent tests that are not as unstable. 21:18:40 the public tests worked for me yesterday 21:18:47 I haven't had any problems recently with the tests 21:18:47 is there something else that i am missing? 21:18:48 cp16net: sweet 21:19:01 i tried them last night w/ a completely fresh install and first run failed w/ like 20 tests failing 21:19:06 2nd run instance never came online 21:19:13 oh but good news 21:19:13 because the jenkins job ran for my code to be merged 21:19:14 stability i think... if you look at most of the patches they have a -1 21:19:18 would it be good to move those tests into the main repo? 21:19:18 12.04.2 networking works again 21:19:20 datsun180b: Could you look at them? 21:19:36 robertmyers: the int-tests? 21:19:40 yes 21:19:43 im gonna have a conversation about the tests w/ ttx too 21:19:47 hub_cap: nice. so should we all be using 12.04.2? 21:19:57 esp1: dont update to it yet i had failing tests 21:20:02 that way we could roll back bad patches 21:20:04 not sure if it was just the flakiness of the env or what 21:20:09 hub_cap: k 21:20:13 Here's a run of the tests where the restart Mysql test failed: https://rdjenkins.dyndns.org/job/Reddwarf-Gate/128/console 21:20:15 i'll blow away my vm and pull everything down tabula rasa 21:20:19 yup robertmyers 21:20:25 one thing i really want to see tho 21:20:33 is that if a test suite needs, say, keystone users, it adds them 21:20:42 rather than redstack 21:20:46 And then passed again on a subsequent run: https://rdjenkins.dyndns.org/job/Reddwarf-Gate/133/ 21:20:48 thats just _an_ example 21:21:03 hub_cap: yes agree 21:21:09 hub_cap ++ 21:21:12 btw in these tests is there a way we can archive the logs so that we can diagnose a little better? 21:21:33 ^ ^ SlickNik 21:21:49 that would be a HUGE help 21:21:57 cp16net: that's something that I can work with Matty on. 21:22:14 awesome should there be a blueprint for that stuff? 21:22:23 i guess it doesnt really apply 21:22:25 #action SlickNik look into archiving logs for rdjenkins test runs. 21:22:25 naw 21:22:29 ya exactly cp16net 21:22:38 another thing is a lot of these pass in fusion but do not in cloud 21:22:43 thx SlickNik 21:22:52 cp16net / others: what logs would be helpful to archive... 21:22:59 the guest log 21:23:03 nova logs 21:23:08 rd logs 21:23:08 reddwarf logs 21:23:12 and rdtest.log? 21:23:14 anything else? 21:23:31 all of the nova logs too 21:23:47 yup, nova logs too 21:23:52 basically anythign thats running, and has a log, and is a openstack project, archive it 21:23:59 (that includes us now :P) 21:24:02 yes sirs 21:24:15 <3 21:24:17 so there is a config for devstack to do this 21:24:19 awesome 21:24:35 robertmyers: oh yeah there is 21:24:36 i think we already use that config robertmyers 21:24:40 in redstack 21:24:53 yeah that just writes the screens to a file 21:24:57 We set the logdir in devstack, if that's what you mean robertmyers... 21:24:59 we need to make sure those are archived 21:25:01 SCREEN_LOGDIR 21:25:02 basically everything is in ./scripts/../report/logs 21:25:03 ? 21:25:27 We just need to copy them to a location that we can access even after the machine goes *poof*... 21:25:32 naw its a different param to get them to log somewhere robertmyers 21:25:33 yeah we had that setup in our ci 21:25:36 we are using _somethign else_ 21:25:44 we could just scp them on to the jenkins? 21:25:47 right SlickNik, jenkins archive works well 21:25:53 archive artifacts 21:26:07 yes vipul, move them to jenkins, archive, keep last, ~10 or whatever 21:26:17 jclouds does all that too ;) 21:26:21 :p 21:26:37 this will be somewhat temporary 21:26:41 Sounds good. Will look into it. 21:26:41 hehe ya 21:26:47 well vipul not really sure 21:26:47 since we'll be moving this to openstack soon 21:26:53 ok moving on? 21:26:53 Yeah, eventually we need to move to devstack-vm-gate. 21:26:55 this is 1off from EVERY other project 21:27:03 so this might stay as is for now 21:27:12 hmm fair nuff 21:27:19 ++ to movin on... 21:27:27 kk, grapex did u have anything to add? 21:27:58 hub_cap: Not really, though locally at Rackspace I'd like datsun180b to look at these tests and try to make them more solid 21:28:06 Spinning up now 21:28:13 awesome thanks 21:28:20 #action datsun180b to look at failing resize and other tests 21:28:37 sweet just making sure u got that action in there :) 21:28:42 #topic Backup status 21:28:51 thanks grapex / datsun180b. That will be sweet... 21:29:03 I want to see logs of your failures to compare 21:29:27 whos? 21:29:30 I'm close to being done with the int-tests for the restore scenario and updating my patch. 21:29:31 datsun180b: I can send you one 21:29:44 for Backup and Restore. 21:29:48 ed.cranford@rackspace.com 21:30:00 datsun180b you could also look at previous failed jenkins run in rdjenkins 21:30:07 ewwwww dont put your email addy on irc 21:30:18 Should be able to get it out for a complete review in the next day or so. 21:30:18 oh no, people know my work email 21:30:19 thats now logged forever on eavesdrop 21:30:31 hah just wait for all that spam to roll in 21:30:35 this channel is not eavesdropped 21:30:40 SlickNik: very cool 21:30:45 i'm no stranger to looking at the logs from rdjenkins 21:30:53 i was doing that last night, recall 21:30:59 vipul: http://eavesdrop.openstack.org/meetings/reddwarf/2013/reddwarf.2013-04-30-21.03.log.txt 21:31:03 u sure bout that ;) 21:31:31 ok so backups is almost ready to go in 1 pr? 21:31:35 err, review 21:31:47 Also need to get eyeballs on: 21:31:48 https://review.openstack.org/#/c/27291/ 21:31:54 #link https://review.openstack.org/#/c/27291/ 21:32:02 #link https://review.openstack.org/#/c/26288/ 21:32:15 #link https://review.openstack.org/#/c/27299/ 21:32:32 how about EVERY review? :-P 21:32:43 that too. 21:32:44 hehe 21:32:47 HA 21:32:54 ya lets all do some reviewing tomorrow / today 21:32:55 yeah we have a backlog of reviews 21:33:02 yup 21:33:06 looks bad on us 21:33:07 SlickNik: Could you run recheck on this one? https://review.openstack.org/#/c/26288/ 21:33:21 will do grapex. 21:33:26 grapex: u mean thru jenkins? 21:33:34 #link https://review.openstack.org/#/q/is:watched+status:open,n,z 21:33:45 Bookmark it, pin it to your homepage 21:33:50 https://rdjenkins.dyndns.org/gerrit_manual_trigger/? 21:33:56 Mark any ones that are work in progress. 21:34:05 Sorry if there's frustration on the fact I haven't looked at some reviews, but there's been some confusing issues. 1) so many builds have failed CI its hard to tell if these PRs are working, and 2) I wasn't certain if the backup API was done or not and which one to start with 21:34:53 retriggered https://review.openstack.org/#/c/26288/ 21:35:17 We talked last time about consolidating the backups pull request, since as I understood it it would have been possible to make a fresh PR from git since the code was finished, and I thought we were going to just do that this one time and then start making smaller ones. Sorry if I misunderstood. 21:35:18 ya the jenkins thing caused us all setbacks i think 21:35:49 grapex: we can review things as they are i think for now. u ok w/ that? 21:35:55 also, did we get int-tests for backups? 21:36:04 id like to see that they are merged when we merge the code itself 21:36:15 sorry for the confusion grapex: https://review.openstack.org/#/c/26288/ should be ready to go. https://review.openstack.org/#/c/27299/ still has the int-tests in progress. 21:36:44 hub_cap: I'll try, since I know there's a need for these things to get merged. 21:36:53 yeah 21:37:01 hub_cap, https://review.openstack.org/#/c/27299/ is WIP since I'm working on the int-tests and that should be uploaded in the next couple of days. 21:37:18 SlickNik: No problem. In the future PM me and let me know your plan behind the order of merging if things get this complicated on a new feature. 21:37:56 grapex: Will definitely keep you up to date. 21:38:00 word 21:38:14 maybe it would be good to outline the plan of attack in the blueprint 21:38:29 i've seen the graphs of deps in blueprints 21:38:33 cp16net: Great idea! 21:38:37 maybe thats how we can mitigate this 21:38:44 ^^+1 21:39:00 +1 21:39:04 i dont know how to create it but it looks really nice 21:39:58 ok so next topic? 21:40:19 Agreed. It's been tricky staying on the same page, so this would be better called out in the blueprint. 21:40:51 also, if while we're in this unknown period, can we all pull down selected reviews 21:40:52 Just to summarize, I think when there are any red marks on a pull request it keeps people from looking at it. In those case we need to be explicit on requesting what we do and don't want people to approve. 21:40:53 #link https://blueprints.launchpad.net/nova/+spec/baremetal-compute-takeover 21:40:54 and run int-tests? 21:40:59 theres kinda an example 21:41:25 grapex: i totally agree w/ u 21:41:33 it moves my eyes away at first glance too 21:41:47 vipul: We could, in this case though the features in a few pieces so its hard to pull it down and just run int-tests. 21:42:17 I kind of think we should all run int-tests independently anyways. 21:42:18 if it helps, we can consolidate the backups patches 21:42:30 because that might be the only big one spanning multiple 21:42:34 esp1: for sure, but its hard to know _fi_ we did that :P 21:42:35 *if 21:42:53 the main problem i see is the two repos 21:42:55 it's just that these tests may continue to fail in jenkins, since it's in-cloud 21:42:58 The issue is that we can't consolidate 21:43:06 Since changes are across repos 21:43:07 why not? 21:43:11 maybe if you see a red −1 from reddwarf you could run the int-tests manually and copy it to the review to show that it works. 21:43:13 vipul: It would, though I already agreed no one has to go to the trouble and would feel bad if anyone had to do that. 21:43:34 we should certainly be running int-tests on our own 21:44:00 esp1: yea that will work for now.. until we get in-cloud tests stabilized 21:44:24 lets also focus on getting them stabilized _now_ rather than later 21:44:31 datsun180b: is our man for that :D 21:44:32 vipul: maybe overkill and not perfect tho 21:44:48 hub_cap: cool 21:45:20 grapex: ok cool, i missed that part of the disucssion 21:45:25 hub_cap ++: we shouldn't let esp1's idea stop us from working on stabilizing the tests. 21:45:35 sure 21:46:04 so that they aren't iffy on an automated run. 21:47:25 is this thing still on? 21:47:30 ok so _now_ moving on? 21:47:36 yup 21:47:37 but to summarize 21:47:41 1) our tests are unstable 21:47:47 2) look @ code anyway 21:47:54 3) check to see if someone has manually run the code 21:48:00 and if u have, put it in the review 21:48:19 1 caveat, _all_ the code, including all of openstack shoudl be up to date when doing this 21:48:22 tahts the only thing that scares me 21:48:39 jsut cuz it works in the microcosm of commit 3ef7aa3 it doesnt mean it works w/ master heh 21:48:49 so keep that in mind 21:48:55 #topic Notifications 21:49:11 If we're going to share results of manually-run tests, I think we need to have some way to basically show a freeze of the components 21:49:16 #link https://review.openstack.org/#/c/26884/ 21:49:22 datsun180b: i think thats fair to say 21:49:25 We can't exactly use pip freeze, but we can share the commit hashes I bet 21:49:40 ya datsun180b 21:49:57 robertmyers: i've been trying to get that to pass reddwarf, but no luck with vm-gate 21:49:58 honestly, in my case if I know it works (having run it locally) I just re-run rdjenkins if I hit an unstable test. It pretty much succeeds on the second run. 21:50:03 can haz merge? 21:50:22 "can haz". Son, people can see you! 21:50:28 ha 21:50:33 robertmyers: is the exists stuff coming in later patch? 21:50:36 hah robertmyers, did u run itn tests 21:50:37 heh 21:50:38 datsun180b: Like that's ever stopped us before. 21:50:40 vipul: what erro? 21:50:45 error? 21:50:58 exists is part 2 21:51:11 this is the start of notifications 21:51:20 robertmyers: https://rdjenkins.dyndns.org/job/Reddwarf-Gate/129/console 21:51:30 basically this is our internal code. 21:51:40 Looks like timeout during mysql stop 21:51:42 hehe the test failed 21:51:44 the exists stuff doesn't exist yet 21:51:49 no can not has merge robertmyers 21:51:58 boo 21:52:14 oh that's 129, I'm still on 128 21:52:32 robertmyers: looks like "test_instance_delete_event_sent" and some of the resize_instance_usage event tests are failing. 21:52:55 yeah, it *should* be using the fake verifier 21:53:28 robermyers: Refresh my memory - that fake verifier actually works in real mode, right? 21:53:29 crap im double booked 21:53:58 Because I remember you had some test doubles (fakes) that worked in fake mode with tox. I'm guessing that actually sends a rabbit event or something you check using rabbit in the VM? 21:54:03 "the fake verifier works in real mode" We are wizards, aren't we 21:54:08 grapex: yes 21:54:31 Ironically this may mean the notifications code is working. :( We've hit issues where they catch bugs for async actions before. 21:54:56 ok im 1/2 paying attn cuz im chatting in #openstack-meeting too 21:55:00 it also needs to have a notification_driver setting in the config 21:55:00 so someone ping me to change topic 21:55:22 datsun180b: I'm more like harry potter in 3rd year 21:55:41 so it could be an out of date reddwarf-integration 21:56:22 clean bill of health from my int-tests 21:56:31 Is there anyway to determine the shas of the git projects that were used in a CI run? 21:56:39 Maybe we should add that to redstack install 21:56:55 https://gist.github.com/ed-/e2947f91ba624595e4d0 21:57:01 just iterate all the /opt/stack directories and save a chunk of the git logs to a file we can archive 21:57:19 grapex: seems reasonable to me 21:57:36 It would solve the "I think this works but I'm not sure if you tested what I tested" problem 21:57:48 Who's with me? Should we make it an action item? 21:57:59 +1 21:58:05 datsun180b: that means its good now 21:58:11 Also an action item: redstack should include more ascii art like datsun180b's VM 21:58:20 that's just the motd of my vm 21:58:37 Starbug is also what a little girl would name a pony. 21:58:37 HEY SlickNik tell them the relevance of the name "Starbug" 21:58:40 Just saying. 21:59:19 woa 21:59:23 starbug is one of the ships in RedDwarf…in case anyone was wondering.. :) 21:59:29 i go to the other meeting room and look what happens 21:59:32 it's Red Dwarf's shuttlecraft 21:59:37 #action keep the crew on topic 21:59:47 #action robertmyers look into failing notifications tests 21:59:53 lol 22:00:19 @action SlickNik to see if there's an easy way to save off the git hashes for a rdjenkins run in case we need to look it up. 22:00:29 #action SlickNik to see if there's an easy way to save off the git hashes for a rdjenkins run in case we need to look it up. 22:00:33 Wait, what is this Reddwarf you speak of? Are you talking about our project, the only thing I know of which ever has used that name outside of astronauts? 22:00:39 protip: it'll involve "git log" and some --flags and some awk i bet 22:00:48 hub_cap: What about the idea to save git histories 22:00:58 so we know if the Jenkins test is testing what we're running locally 22:00:59 ? 22:01:03 vipul: ^^ 22:01:09 I hear you datsun180b, pretty much what I was thinking. 22:01:12 i mean 22:01:17 i dont know if we need to go that far 22:01:21 whos gonna verify it? 22:01:25 just be sure your up to date 22:01:28 hub_cap: That's not the point 22:01:34 then what is the point? 22:01:40 record keeping 22:01:41 vipul ain't at his desk atm 22:01:44 The point is to be a sanity check so if Jenkins is failing, and you're positive it's working locally, you can know if the code is the same. 22:02:03 #agree 22:02:07 Because maybe you make a mistake and are on a different version of RDI, or the client, or Nova, or some other thing than Jenkins is using 22:02:17 you know that could be some "change log" which we are lacking in the public now 22:02:24 you hush 22:02:32 cp16net: We need that too but this would be simpler 22:02:40 ok my Q is this 22:02:41 sure but it could be one in the same 22:02:43 whos gonna use that info 22:02:45 i think we need releases to have changelogs 22:02:47 or at least a first step 22:02:58 are u gonna look @ the SHA's if someones stuff doesnt run on CI? 22:03:03 or if htey claim it does run locally 22:03:13 are u gonna validate all 9 of the SHA's against whats in master for all of openstack? 22:03:21 hub_cap: That doesn't matter 22:03:32 we'd look at them in the case of a discrepancy 22:03:38 all you do is say "Hey, it works locally! What are my shas?" Check them, then "ok, what are the Jenkins shas", check them 22:03:39 ok lets take this off to the regular room, cuz we are already at time 22:03:42 And move on with your life 22:03:52 hub_cap: I'm more interested across our 3 projects than across openstack. 22:03:58 But I see your point. 22:04:38 Right now what I'm reading is robertmyers has tested his code in the VM and thinks Jenkins isn't running the same code. If we had the shas archived it would be trivial to verify if that was the case. 22:04:45 #agreed hub_cap 22:04:55