22:00:56 #startmeeting qa 22:00:57 Meeting started Thu Jul 24 22:00:56 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:00 The meeting name has been set to 'qa' 22:01:07 hi, who do we have here today? 22:01:18 o/ 22:01:21 hi 22:01:29 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_July_24_2014_.282200_UTC.29 22:01:31 hi 22:01:32 hi 22:01:34 ^^^ Today's agenda 22:01:59 o/ 22:02:10 ok well let's get started 22:02:18 hi 22:02:23 #link Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish) 22:02:29 d'oh 22:02:36 #topic Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish) 22:02:55 so I won't be able to make the next 2 meetings 22:03:07 mtreinish: I can do next week but will be out Aug 7th 22:03:11 so I'm looking for a volunteer to run them and send the reminder email out 22:03:33 dkranz: ok, I think having one person do both would be preferable but we can split it up 22:03:43 especially given the timezone differences 22:03:52 I can't join next 2 meetings.. sorry. 22:04:03 I also cannot make both :-( 22:04:25 both the next meetings are iffy for me actually 22:04:37 masayukig, oomichi: heh, stop doing that, you're beating me to asking you if you can lead the Aug 7th meeting :) 22:04:53 hmm, ok well if no one can run the Aug 7th one then how about we just cancel it 22:05:12 so dkranz if you want to take July 31st 22:05:18 and I'll cancel the Aug 7th meeting 22:05:28 +1 22:05:32 mtreinish: sure. If there is no one around to run the meeting it won't be very useful :) 22:05:58 #info dkranz to run July 31st meeting and send email reminder 22:06:05 #info Aug 7th meeting is cancelled 22:06:12 ok let's move on to the next topic 22:06:21 #topic Mid-cycle Meetup Summary (mtreinish) 22:06:36 so last week we had the midcycle meetup in Darmstadt 22:06:42 which I thought was very productive 22:06:49 #link https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014 22:06:59 that's the etherpad we were using to track things during it 22:07:30 but the work items that started out of it were 22:07:43 the test-accounts alternative to tenant isolation 22:08:02 converting the scenario tests to use the tempest clients 22:08:16 and the giant re-architect the gate thread 22:08:26 which jeblair and sdague pushed out to the ML 22:08:53 yeh, that would be great to get folks in on if they haven't yet looked at it 22:09:13 I just wanted to give a brief overview of what we were working on for those who couldn't attend 22:09:28 so unless there are any questions I think we can move on 22:10:23 #topic Blueprints 22:10:34 ok so this week is the J-2 milestone 22:10:49 so I wanted to go over the BPs targetted for J-2 and see what's going on for those 22:10:56 #link https://launchpad.net/tempest/+milestone/juno-2 22:11:28 masayukig: heh, so for Migrate the Scenario Tests to the Tempest Clients we should probably defer it to J-3 right? 22:11:44 we probably were a bit overoptimistic last week when we started it :) 22:11:57 mtreinish: yeah, that's right. 22:12:18 ok next on the list is javelin 2 22:12:19 mtreinish, masayukig: yes - but that's moving forward quiclky at least :) 22:12:29 sdague: that's a defer too right? 22:12:48 andreaf: yeah, it is moving quickly, which is great 22:12:48 mtreinish: yes, we're close, but it's definitely going to defer 22:13:00 sdague: ok 22:13:50 hmm so does anyone know Arthur Svechnikov's nick? 22:14:15 although I'm pretty sure ceilometer scenarios is also a defer 22:14:33 andreaf, mtreinish: yeah, 10/18 test cases are done or doing, now. 22:14:49 #link https://etherpad.openstack.org/p/tempest-client-scenarios 22:14:53 masayukig: yeah that is a lot of progress for 1 week 22:15:17 masayukig, mtreinish: I'm trying to get onboard some more collegaues on it 22:15:20 hmm looking at the rest of the list I think all the other bps are defers too 22:15:25 mtreinish: Isn't your "Add a config file verification tool" "done" ? 22:15:46 dkranz: well it still needs some more testing coverage 22:15:48 mtreinish: I mean I am refactoring it right now but that is not really related to the blueprint, more to the one from boris 22:15:59 but functionaly I think I've gotten most of down already 22:16:01 mtreinish: ok 22:16:08 dkranz: but sure I can close it out 22:16:12 mtreinish: I suggest waiting until I check in the refactor for that. 22:16:19 unless there are other discoverable options I've missed 22:16:27 dkranz: ok that's fine I'll defer it too 22:16:29 mtreinish: If there are, I will find them. 22:16:55 mtreinish: ANd include them in the refactoring 22:17:02 dpaterson: you joined right in time :) 22:17:16 Sorry I was late, traffic 22:17:20 dpaterson: any update on https://blueprints.launchpad.net/tempest/+spec/post-run-cleanup 22:17:25 andreaf: great! 22:17:27 since the j-2 milestone is this week 22:17:28 yes 22:17:52 I have most of it done. The last piece I am working on is the --dry-run. 22:17:59 I am going to finish that up early next week 22:18:08 dpaterson: ok have you pushed anything up for review yet? 22:18:19 it's ok to do it iteratively 22:18:23 Not yet 22:18:40 dpaterson: ok, then I should defer it to the next milestone? 22:18:50 When is that? 22:19:21 dpaterson: Sept. 4th I believe 22:19:38 yes, that seems like a better plan 22:19:47 dpaterson: it's not going to get merged this week so yeah :) 22:19:51 dpaterson: deferring it doesn't mean you can't check it in as soon as you want :) 22:19:58 yes 22:20:16 ok the last thing is mriedem's BP for the CLI version check 22:20:19 I can push code pretty soon. 22:20:25 dpaterson: ok awesome 22:20:41 that BP should be finished after https://review.openstack.org/100031 lands 22:20:54 so we can close that out on schedule if it get's merged this week :) 22:21:26 but if not it's no big deal 22:21:53 ok aside from the people who I know aren't here or I don't know their irc handles that should cover all the BPs for J-2 22:22:11 I'll reach out to those I missed outside the meeting 22:22:16 but I think they're all defers 22:22:27 so does anyone have anything else to discuss on Blueprints 22:22:32 otherwise we can move on 22:22:36 mtreinish: how is that BP related to the cleanup work I am doing? 22:22:51 dpaterson: the BP is used for tracking that effort 22:23:20 the spec proposal you got merged, turned into the blueprint so we can track it's progress 22:23:26 Add min_client_version decorator for CLI tests? 22:23:38 That is the title 10031 22:23:57 dpaterson: oh, sorry I jumped the gun a bit and changed the BP we were discussing before you were finished 22:24:19 it was a separate thought from your cleanup BP discussion 22:24:25 no worries 22:24:39 making sure I wasn't missing something 22:24:41 tx 22:24:50 ok if there's nothing else then let's move on 22:25:00 #topic Specs Review 22:25:12 so I put one spec in the agenda this week: 22:25:19 #link https://review.openstack.org/#/c/108858/ 22:25:43 I basically outlined how I think we should break out a portion of tempest's common code to create an OpenStack functional testing library 22:25:57 this can be re-used for project specific functional testing 22:26:10 which was a component of the ML thread on changing what we do in the gate 22:26:44 It's a little sparse on the details right now 22:26:55 mtreinish: LGTM except for my comment 22:26:57 but if people could take a look and add their thoughts that would be great 22:27:09 dkranz: yeah I have a response drafted I just need to finish it 22:27:10 mtreinish: I'll add it to my review list 22:27:23 mtreinish: are you thinking of a new olso.xyz 22:27:40 andreaf: I specifically didn't want it in oslo, because I want it to be under the QA program 22:28:04 I also wanted to call it mesocyclone, and oslo.mesocyclone just looks weird :) 22:28:35 andreaf: It would also be weird for oslo to have another rest client 22:29:01 ok that's all I had for my spec review 22:29:16 does anyone else have a spec proposal that they would like to bring up? 22:29:17 Some people already wonder why it is not unified with the official client 22:29:44 the test-account spec, what shall we do? 22:29:58 dkranz: yeah, that I don't know, I think it comes down different implementation goals 22:30:01 #link https://review.openstack.org/#/c/98400/ 22:30:18 andreaf: oh, that's a good point I forgot to review that before we started working on it 22:30:29 mtreinish: I don't either but it was discussed at the neutron mid-cycle 22:30:53 andreaf: I'll take a look at it tomorrow and fix anything that differs from what we've implemented and self approve it 22:30:58 if no one objects to that 22:31:21 mtreinish: sure I'm ok with that 22:31:34 anderstj: There was a -1 review with some nits 22:31:45 andreaf: ^^^ 22:31:51 anderstj: Sorry 22:32:18 +1 for me on eventual unification 22:32:28 I think there could be a core that encompasses the goals of the tempest client 22:32:36 dkranz: I'll have a look as well - mtreinish: if you want to post comments I can make a new version as you wish 22:32:38 and something that uses that core but with end-user niceties 22:32:51 marun: heh, I'm not sure that was actually on the table :) 22:32:58 I know it's not on the table now 22:33:03 that's why I said 'eventual' 22:33:14 andreaf: I also will join to the spec. 22:33:15 marun: heh, fair enough 22:33:16 when the unnecessary duplication becomes painful and therefore obvious 22:33:21 :) 22:33:51 ok are there any other specs to discuss? 22:34:02 marun: I think trying to make that now would divert us from the primary focus to much 22:34:15 marun: but eventually it may be 22:34:25 andreaf: yeah, I'm not pushing hard for thinking about it now 22:34:31 just planting a sead 22:34:33 seed 22:35:10 ok let's move to the next topic 22:35:18 #topic Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz) 22:35:33 dkranz: with the absurdly long topic title, you've got the floor 22:35:37 mtreinish: So this was based on two opinions I have 22:35:51 1. We are paying too much for "double-entry" bookkeeping 22:36:08 2. It has never been clear exactly what the content of tempest api tests should be 22:36:44 Right now they are not enough to keep out bugs, but enough to run a long time and trigger flaky bugs 22:36:58 One good answer is more func tests in projects 22:37:29 But having a full set in tempest does not scale 22:37:36 Open for other comments :) 22:37:39 dkranz: 100% agree 22:37:53 we've grown beyond the point where that's sustainable 22:38:12 sdague: So I am proposing something very simple for now 22:38:28 I actully think this was the original purpose of the smoke tag 22:38:44 if moving API tests to each project, will tempest contain only scenario tests and CLI tests? 22:38:45 It was never envisioned that tempest api would get to where it is now 22:38:56 It grew organically and with the tc command for tempest tests 22:39:08 oomichi: NO, it needs to have cross-proejct api tests 22:39:20 oomichi: The trick is that it is not always obvious how to determine that 22:39:20 dkranz: I agree too, but I don't think we should be running less in the gate now. Just not grow it much and start to migrate things to the projects as functional tests 22:39:27 oomichi: But I think we should trty 22:39:31 dkranz: aka integration tests 22:39:42 oomichi: actually cli tests will be the first thing converted to project functional tests 22:39:59 mtreinish: I am not really opposed to that but we have to define what is supposed to be there 22:40:01 I would hope that the goal would be migrating responsibility rather than hack/slash, so that we don't lose out in the transition. 22:40:09 oomichi: because there isn't anything that's cross-project about them, they really are functional tests, not integration tests 22:40:32 oomichi: so honestly, I'd start with migrating out the CLI tests 22:40:32 mtreinish, dkranz: ah, I can see it. 22:40:41 Question: Most of Refstack tests are API tests. Could QA still be the central agent (at least one +2) while the tests get executed outside of tempest? 22:40:59 marun: yes, which was basically my comment on the ML thread about the gate rearchitecture 22:41:10 mtreinish++ 22:41:14 I don't want to see a drop in coverage while we move to something more targetted 22:41:15 marun: we're not rushing this one 22:41:29 I think QA still needs some ownership, including repo, just not the infra the tests run on. 22:41:35 this is about building momentum towards a better system 22:41:37 tracking which api we touch in either API or scenario tests may help us getting an idea of the kind of coverage we have 22:41:52 mtreinish: fair enough. But we have to define what will be in tempest going forward 22:41:55 it would be good to have something like the stress api tracker for all tests 22:42:12 dkranz: yes I agree, we need to figure that out before we start doing anything 22:42:16 I think in an ideal case the projects write new API and scenario tests in-tree, and then migrate into separate repos once they have been stabilized. 22:42:17 Could API test results feed into QA the way refstack results feed into Refstack.org? 22:42:50 mtreinish: And there would be a lot of value in running only nova smoke against other projects. Whether that value has greater costs is debatable. 22:42:53 Whether QA or qualified project-specific reviewers maintain those 'test' repos would be an implementation detail. 22:42:54 marun: or just put a hacking freeze on those tests 22:43:15 marun: yes I think something like that would be an ideal way to do it, but we need criteria on what kind of test is allowed to do that 22:43:19 and what should be in tempest 22:43:23 sdague: I'm not sure what you mean, how would it protect against accidental modification? 22:43:38 marun: I think he's saying have the pep8 job fail if the test is changed 22:43:53 mtreinish: I didn't realize that was possible, link? 22:43:54 marun: enforce a rule where some part of the tree can only be changed if it's the only part that's changed 22:44:20 marun: that code doesn't exist yet, but it would be a thing one could write 22:44:37 That sounds great if it would allow us to protect API tests against regression. 22:44:46 sdague: ah, gotcha. Definitely a worthy initiative 22:44:51 marun: yeh 22:45:29 honestly, my goal here is to move more responsibility back to the projects, and hopefully help expose some issues in a simpler state of the world that means better overall openstack quality 22:45:30 sdague: yes, that would be a less costly way to get double-entry bookkeeping 22:45:36 dkranz: yep 22:45:51 sdague: ++ 22:46:08 sdague: ++ 22:46:09 sdague: Let's think about what tests tempest would actually have in that world 22:46:31 sdague: you sure flip flop on your opinion about hacking :) 22:46:49 dkranz: so I think give our time constraints we'll have to take that converastion outside the meeting 22:46:52 isn't it better to be right than hypocritical? ;) 22:46:56 mtreinish: agreed 22:47:04 that made no sense, sorry 22:47:12 mtreinish: I just wanted to get this discussion going as a follow-on to Sean and Jim's email 22:47:25 hypocritical + right vs consistent + wrong 22:47:43 marun: it's called learning and adapting :) 22:47:56 dkranz: yeah I plan to start a separate ML thread in response to: http://lists.openstack.org/pipermail/openstack-dev/2014-July/041182.html 22:47:59 sdague: ++ 22:48:23 although I think we'll inevitably have a summit session titled: "Tempest Scope in the Brave New World" 22:48:33 ++ 22:48:38 mtreinish: +1 22:48:41 mtreinish: great. I feel like we have turned a major corner here to make things better 22:48:46 Mtreinish: +1 22:49:10 mtreinish: nice title:) 22:49:24 mtreinish: ++, look forward to it 22:49:41 but, I definitely don't want to punt on the discussion until Nov., so I think we should definitely take this to the ML and other places in the mean time 22:49:55 anyway we've got 10min left so I think we should move on 22:50:04 mtreinish, sdague: the post gate testing part of it opens up space for more complex and possibly long running tests as well - which could live in tempest in the brave new world 22:50:38 andreaf: +1 integrated tests, scenario tests, etc 22:50:49 #topic Critical Reviews 22:50:55 andreaf: maybe... 22:51:06 sdague :D 22:51:16 ok does anyone have any reviews they'd like to bring up? 22:51:27 mtreinish: all the reviews related to scenario migration 22:51:34 andreaf: a link? 22:51:36 especially the base onse 22:51:45 andreaf: honestly, getting more debugable functional tests would be might better for overall quality 22:51:49 #link https://review.openstack.org/107552 22:52:10 #link https://review.openstack.org/107562 22:52:56 #link https://review.openstack.org/107371 22:53:04 sdague: perhaps... but if you can collect big data (using subunit2sql) and watch trend on less stable tests it may give you something interesting 22:53:22 that is also from the mid-cycle and I think it is ready since I added more unit tests 22:53:42 :) 22:53:55 mtreinish: I'll check it tomorrow 22:54:18 andreaf: well, I need to get subunit2sql into infra first :) 22:54:19 pre-unit tests it was good already for me 22:54:38 andreaf: heh, well I found some bugs through unit testing... 22:54:46 ok are there any other reviews to bring up? 22:55:40 ok then let's move on 22:56:04 #topic Proposal of temp review policies (andreaf) 22:56:23 there are three things I wanted to propose 22:56:48 New scenario tests shall use tempest client. I guess people could rebase on in-flight changes? 22:57:03 until the migration bp is completed 22:57:12 hmmm 22:57:24 so I'm fine with that one, except we have some long standing reviews adding scenarios 22:57:35 I'd hate to churn on that again to rebase and change the client usage 22:58:01 I'm thinking mostly around heat 22:58:03 mtreinish: ok - well we left the original base classes in for now 22:58:29 mtreinish: yes I started working on the existing heat ones, but it may take a bit longer 22:58:42 andreaf: but for submissions that are newer than last week I definitely agree 22:58:53 andreaf: < 2 min so what are the other 2? :) 22:58:58 mtreinish: ok makese sense 22:59:04 New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently 22:59:08 New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest 22:59:25 so we actually have an agreement on that last one from several meetings back 22:59:32 I can dig up the logs 22:59:34 ok 23:00:08 the second one I'm not sure about, mostly because I'm not sure what the goal is 23:00:10 I though the last proposal was to make the job voting on neutron only but I think it should be on tempest as well 23:00:28 ok we can take it offline time's up 23:00:33 andreaf: yep 23:00:38 ok thanks everyone we're at time 23:00:42 #endmeeting