22:00:17 #startmeeting qa 22:00:19 Meeting started Thu Mar 20 22:00:17 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:23 The meeting name has been set to 'qa' 22:00:34 hi who's here today? 22:00:37 hi 22:00:38 hi 22:00:45 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:00:49 ^^^ Today's agenda 22:00:58 hi 22:00:59 hi 22:01:32 hi \o 22:01:59 sdague, dkranz: you guys around? 22:02:05 well let's get started 22:02:06 Yes, just here 22:02:17 #topic Blueprints 22:02:18 o/ 22:02:44 so sdague I'm assuming you left the note on the agenda about the specs repo 22:02:52 yes 22:03:12 I wanted to make sure that we started talking about the qa-specs repository 22:03:18 #link https://github.com/openstack/qa-specs 22:03:55 so based on the conversation last week we're going to try to do something very similar to what nova is doing with a -specs repository for reviewing blueprints 22:04:21 I think the right thing to do is unapprove all non-high items, and have specs proposed back in through this new process 22:04:40 that will hopefully help us actually clarify blueprints in advance 22:04:54 yeah I agree that's probably the right approach for the exisiting bps 22:04:56 a couple of questions for folks 22:05:18 1) do you think we need a more formal template? or is a free form better to get started? 22:05:36 I am leaning towards free form, as I'd like to build template by experimentation 22:05:42 sdague: I'd say free form 22:05:48 however, if people feel strong the other way we can do that 22:05:54 maybe a very rough template for some guidance? But I'm happy either way 22:05:56 Until we get a feel for what people will put there 22:06:10 sdague: yeah I think free form is better too, but maybe some guidelines on what content is expected in the proposal 22:06:35 sdague: for example our debate on where to put target release 22:06:42 cyeoh: you have some ideas about what rough guidance? 22:06:49 can probably steal the basic stuff from the nova template - things like pointing back to the blueprint etc 22:06:55 ids of who is going to work on it etc 22:07:09 cyeoh: ok, sure, that's probably fair 22:07:30 cyeoh: you want to write a few things here - https://etherpad.openstack.org/p/qa-spec-template 22:07:34 I can do that this afternoon if you'd like (just can't do it this morning) 22:07:46 cyeoh: sure 22:08:03 so if you are going to do it your afternoon, yuo want to just propose it as a review to the repo? 22:08:06 and we'll do it that way 22:08:24 sure, will do 22:08:58 I think basic problem statement, and approach, target time frame, people working on it, and where it would go in the tempest tree are the things I can think of at the moment 22:09:06 my brain is a bit gate fried though :) 22:09:19 sdague: yeah that seems like a reasonable list 22:09:21 so the assistance on proposing template is appreciated 22:09:23 so are we planning on having people resubmit every cycle? 22:09:34 cyeoh: I don't think so 22:09:41 ok 22:09:55 I think if an idea goes obsolete then we mark it somehow in the template 22:10:01 sorry, in the spec 22:10:15 but I think we can figure out that workflow later 22:10:20 should we stop unapproved bp tasks until approval with this process? 22:10:29 sdague: sounds good to me 22:10:52 ken1ohmichi: honestly, I don't want you to stop doing anything :) 22:10:55 you're too productive 22:11:03 +1 22:11:09 +1 22:11:28 thanks:-) 22:11:32 mostly I want us to go back through this process to make sure things are clear 22:11:44 for juno 22:11:59 because we definitely got into a lot of confusion on things like the advanced network debugging 22:12:12 that I feel like needs an overview document to make sure we're on top of it 22:12:39 so consider this an "in parallel" for anything you are actively working on 22:12:49 and a pre req for new things 22:13:07 sdague: So should be be more accepting of proposals that are project-specific? 22:13:24 sdague: Both neutron and heat wanted to have more blueprints in tempest but we said no 22:13:36 yes, I feel like doing this I'd be ok with that 22:13:50 sdague: Yeah, we should at least be open to it 22:14:29 because we don't end up with them populating 50 lp artifacts and no one noticing :) 22:14:52 and important part of this process is "Bring forward the proposed item to the openstack-qa meeting for summary" 22:15:16 and I don't want us +Aing anything that misses that step 22:15:18 sdague: Right. And there could be project subdirs in the specs area 22:15:29 dkranz: maybe, lets try flat first 22:15:43 sdague: k 22:15:47 we can always easily reorg if it's gone crazy 22:15:55 dkranz: I'm still not convinced a bp or this repo are the best way to track doing test tracking either 22:16:00 so there may be some people who find it really hard to get to the irc meeting 22:16:01 but we can iterate and explore 22:16:08 are we ok with people just putting it on the agenda if that's the case? 22:16:14 (and not turning up) 22:16:25 mtreinish: Yes, since real project management tools are out-of-bounds :) 22:16:36 cyeoh: even with the rotating times? 22:16:44 yeah that's fine. I guess we don't have coverage for certain tz even with the rotating sched 22:16:48 sdague: yeah like china 22:17:04 right that's 6am now? 22:17:17 yep 22:17:45 well, one of the issues I really want to address is people off working on bp that aren't communicating via irc or email about them 22:18:19 because we definitely have seen people not be successful if they only communicate in gerrit 22:18:33 and I'd like to have people be more successful 22:18:34 sdague: yeah I agree but if the meeting can't be attended by everyone we need some way to enforce communicate besides just the meeting 22:18:51 sure, so alt to meeting is ML discussion 22:19:03 mtreinish: We could insist on initial ml post 22:19:16 well, lets see how the first round goes 22:19:17 ok that works 22:19:31 sdague: do you think we should update the readme 22:19:34 lets start with all of us that have bp that are < high proposing up stuff 22:19:36 because it doesn't mention the ml 22:19:39 in the next week 22:19:55 then out of that set we can start to figure out what's workign and what's not 22:20:31 so we'll do a review next week of everything that's been proposed and comment on what seems to be working and not with the repo 22:20:36 sound good to folks? 22:20:41 +1 22:20:42 works for me 22:20:57 +1 22:21:02 +1 22:21:03 +1 22:21:04 +1 22:21:13 after we have that, we can communicate more the the broader community to get more folks doing it as well 22:21:24 hopefully that will provide plenty of good examples for them to copy from 22:21:55 +1 22:21:56 ok, I think that topic is done unless there are last questions 22:22:35 * afazekas is reading more frequently IRC than ML 22:22:42 ok then let's move on to the bp review 22:23:06 I think for time we should skip the standard high prio bp walkthrough 22:23:17 sure 22:23:27 and just ask if anyone has any updates for open bps 22:23:37 or needs feedback or attention on any bps 22:24:00 #link https://blueprints.launchpad.net/tempest/+spec/unit-tests 22:24:23 I think I'll just mention that we've made a bunch of progress with unit tests this past week 22:24:36 we've had some new contributions from masayukig and others 22:24:38 question on the unit tests, in fixing something for the logging 22:24:54 we seem to be using a lot of fake objects 22:25:02 instead of just mock calls directly 22:25:08 is there a reason for that? 22:25:31 sdague: there are a few fake objects for some common things that will be used everywhere 22:25:39 but for the most part it should be more mocks and fixtures 22:26:02 for example look at the isolated_creds tests 22:26:25 anyway I think we'll be at >200 unit tests by the end of icehouse 22:26:30 yeh, I would lean on trying to get rid of the fake objects if possible, mostly because it adds a level of coupling when you need to change them 22:26:37 mtreinish: nice 22:26:38 considering we had <10 at the start 22:26:54 sdague: yeah some of it was me just being lazy at first 22:26:59 and it being reused 22:27:26 we can start to clean that up over time (like the config refactor to use the fixture) 22:27:35 yeh, cool 22:28:16 honestly, I've only really started wrapping my head around mock recently, but it's become the openstack standard, so we should use it more to make it easier for people to jump between projects 22:28:21 all goodness 22:28:36 yeah I agree 22:28:46 ok I didn't have anything else to mention about unit tests 22:28:52 does anyone else have a bp to bring up 22:28:56 otherwise we can move on 22:29:30 #topic Neutron testing 22:29:51 mlavalle: are you around? 22:29:55 hi 22:30:22 mlavalle: any update on neutron testing? 22:30:23 We have continued merging api tests. Since our last meeting, another 2 patch sets have merged 22:30:41 The grand total over the past 4 weeks is 14 22:31:32 There are 3 abandondes patch sets. I contacted their authors by mail. One of them responded (he owns 2) and has given me permission to assign the patch sets to someone else 22:32:13 I also have 3 patch sets that only rehire one additional +2 to merge. Can you guys help? 22:32:21 mlavalle: sure do you have links? 22:32:33 yes, give me aminute 22:33:09 https://review.openstack.org/#/c/71251 22:33:09 https://review.openstack.org/#/c/68597 22:33:10 https://review.openstack.org/#/c/63999 22:33:28 #link https://review.openstack.org/#/c/71251 22:33:32 #link https://review.openstack.org/#/c/68597 22:33:37 #link https://review.openstack.org/#/c/63999 22:33:43 In summary, continuing making good progress 22:33:47 mlavalle: ok I'll take a look tomorrow morning 22:33:53 but someone will probably beat me to it 22:33:55 thanks :-) 22:34:00 That's all I have 22:34:13 ok cool thanks 22:34:29 I guess we can move on if no one has any questions about neutron testing 22:35:08 #topic Heat testing 22:35:30 sdague: I added this to the template 22:35:35 but who should I ping about it 22:35:59 stevebaker was the primary one code was coming from 22:36:18 \o 22:36:24 and there he is! 22:36:32 lefty and all 22:36:37 stevebaker: hi, do you have any updates on heat testing with tempest 22:37:24 I've refreshed the autoscaling test, its working locally https://review.openstack.org/#/c/44967/ 22:37:34 stevebaker: cool 22:37:41 but it is waiting on firewall changes to land before it can run https://review.openstack.org/#/c/81375/ 22:38:14 efforts to shame other heat developers to write tests may yet yield some results ;) 22:38:26 stevebaker: I was trying to pull apart what would be needed for an "ha" test using heat 22:38:43 where should I dig in to learn enough to do that? 22:38:44 I hope to start writing some software-config tests once a few fixes land 22:39:05 basically "restart vm if it goes away", which I gather we can do from heat 22:39:53 sdague: the autoscaling scenario test can continue to evolve so that it scales arbitrary stacks, and is fronted by a neutron load balancer 22:40:06 sdague: most likely as new tests 22:40:12 i mean new scenarios 22:40:18 stevebaker: sounds good 22:40:37 maybe I'll ping you in -dev tomorrow to ask some more questions about how to help here 22:40:41 I'll be focusing on software config though 22:41:13 sdague: ok, thanks 22:41:34 stevebaker: ok cool, is there anything else? 22:42:14 we need some stack update scenarios, but it remains to be seen who will write those 22:42:52 stevebaker: maybe a call for help on the ML? 22:43:51 mtreinish: maybe. Existing developers know the tests are needed, its just a matter of resourcing it 22:43:58 cool 22:44:20 look forward to more here. It's definitely good to have the heat job voting on everyone now 22:44:37 yes, it seems solid 22:44:49 yep 22:45:25 ok, next up? 22:45:28 ok if there's nothing else to discuss let's move onto the next topic 22:45:34 #topic Bugs 22:45:47 so we had our bug day yesterday 22:45:55 maurosr: do you have any results from it 22:45:57 ? 22:46:02 yup 22:46:14 so just the numbers to be clear 22:46:26 bugs that needed triage from 62 to 18 http://bit.ly/1cXhPcF 22:46:35 yay! 22:46:36 ok 19 now (just have a new one) 22:46:40 Open bugs from 166 => 146 22:46:48 To prioritize we kept on 29 (no evolution) 22:46:54 In progress 51 to 57 http://bit.ly/1hzsjfH (which is kind of result of taking triage bug into action) 22:48:07 I tended to move some of the bugs to Incomplete instead on invalid giving a chance to the reporter make it clearer... that is why we reduced only in 20 the open bugs 22:48:28 sure, that sounds fair 22:48:34 thanks for driving on this one 22:48:41 yw 22:48:41 +1 22:48:45 maurosr: Yes, thanks 22:49:13 btw 22:49:37 in one of our meeting we were discussing thwe kind of stuff that we would accept as bugs 22:49:44 not only traces 22:50:03 yeh, if it's just a trace I was pretty aggressively marking as invalid 22:50:10 as it's almost never a tempest bug 22:50:29 so it happens people are using tempest bugs to do rechecks, so there are lots of bugs that should at least affect other projects too that were there 22:50:41 sdague: But don't we want to get these things into elastic recheck? 22:50:44 there was even a cinder unit test bug 22:51:09 dkranz: it needs to be more than a stack trace for ER 22:51:10 maurosr: heh, yeah I saw that one that cracked me up 22:51:24 and it really shuold be assigned to the project where the problem is 22:51:35 and delete tempest from the bug if it's not actually a tempest bug 22:51:42 sdague: I'm fine with that rather than just marking it invalid 22:51:51 because if the ER bugs aren't actionable 22:51:55 then they aren't useful 22:52:00 sdague: Let the projects mark them as invalid :) 22:52:07 invalid != remove tempest of the affected list? 22:52:25 maurosr: so honestly either is fine 22:52:35 maurosr: if there is more than one project listed on the bug you can remove one from the list 22:52:55 right, understood what you meant 22:53:09 yeh, I tend to remove the wrong projects 22:53:27 because I find that otherwise people get confused about the bug, especially if they reopen it later 22:53:43 also, feel free to rewrite the summary and title to be more descriptive 22:53:49 if the base bug is bad 22:53:58 that helps create clarify over time 22:54:32 this is really policy for anyone, not just maurosr :) 22:54:38 heh 22:54:50 maurosr: ok is there anything else on bugs? 22:54:51 but especially bugs that we want to put into ER, I like to have good summary line 22:55:01 nop, I will send a summary to the list later, but I guess related to the bug day it's pretty much it 22:55:12 ok 22:55:21 let's move on then 22:55:28 5 minutes 22:55:28 #topic Critical Reviews 22:55:41 does anyone have any reviews they'd like to get eyes on? 22:56:18 if nothing else - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:rest_client_logging,n,z 22:56:29 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:rest_client_logging,n,z 22:56:31 I'm trying to tighten up the tempest log so it's useful 22:56:48 which means dumping a lot of the noise from it 22:56:57 For the leak related change I need to xml clients to behave more closely to the json clients https://review.openstack.org/#/c/81847/ https://review.openstack.org/#/c/78345/ 22:57:04 not complete, but that definitely cleans it up a bunch 22:57:11 #link https://review.openstack.org/#/c/81847/ 22:57:17 #link https://review.openstack.org/#/c/78345/ 22:57:33 sdague: yeah it's definitely a step in the right direction 22:57:51 I even killed the cli output on success in the last patch 22:57:54 although with os-log-analyzer there is too much blue right clumped together with the patch 22:58:13 heh 22:58:25 well we can debate colors at the next summit 22:58:31 I could unbold info as well 22:58:40 I think that would probably be enough' 22:58:51 yeh, I can propose that tomorrow 22:59:10 Looks like the instance validation with n-net with HARD reboot is flaky https://review.openstack.org/#/c/81834/ :( , at my home desktop the test seams stable. 22:59:39 I am looking for advises how to move forward 22:59:59 ken1ohmichi: ^^^ that seems like it's in your court 23:00:05 anyway we're out of time 23:00:07 thanks everyone 23:00:22 I'll add the agenda item we couldn't get to onto next week's 23:00:25 mtreinish: Can you move my agenda item to next week? 23:00:25 #endmeeting