17:00:10 #startmeeting qa 17:00:11 Meeting started Thu Mar 13 17:00:10 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:14 The meeting name has been set to 'qa' 17:00:21 o/ 17:00:26 hi who's here today 17:00:34 hi everyone 17:00:50 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_March_13_2014_.281700_UTC.29 17:00:55 hi 17:00:57 ^^^ today's agenda 17:01:30 o/ 17:01:43 ok, lets run through the high priority blueprints 17:01:52 #topic Blueprints 17:01:56 yes, let's get started 17:02:03 sdague: do you want to roll through the list? 17:02:06 sure 17:02:10 #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full 17:02:38 Hi all ;) 17:02:46 rossella_ put out details on the list about the bugs found 17:03:16 I'm going to mark that good progress, and hope it gets closed soon 17:03:37 #link https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests 17:03:55 andreaf: where do we stand there? there is another review out, right? 17:03:55 sdague: so I split my patchsets further 17:03:59 ok, great 17:04:06 #link: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z 17:04:18 I have 5 patchsets there now 17:04:32 plus I have two patchsets for getting the experimental keystone v3 check 17:04:38 ok, cool 17:04:50 #link https://review.openstack.org/#/c/79212/ 17:04:56 I think that remains good progress then 17:05:00 #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests 17:05:00 #link https://review.openstack.org/#/c/79314/ 17:05:28 I just marked v3 api tests as implemented. I think cyeoh is going to open a new blueprint for juno 17:05:38 yeah that's what we discussed last week 17:06:06 #link https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration 17:06:14 we have heat-slow voting now, which is good. 17:06:51 if stevebaker is around we could figure out if we should call that done, and create a new BP in juno, or move that one to juno. 17:07:03 or anyone else that can speak for heat 17:07:19 it looked like they were talking about a tempest day, or set of days in the heat meeting yesterday 17:07:45 so I'd like to make sure that we consider new heat patches high priority for review in tempest to support them in that 17:08:06 #info consider tempest patches for heat functionality high priority from review perspective 17:08:09 do you want to info that? 17:08:20 oops you win 17:08:22 heh 17:08:26 #link https://blueprints.launchpad.net/tempest/+spec/unit-tests 17:08:28 sdague: I think this has been the case for a while but there have not been many patches 17:08:34 dkranz: agreed 17:08:39 I'm hoping that changes 17:08:45 sdague: RIght 17:08:55 mtreinish: how do you want to handle the unit tests blueprint 17:08:59 so unit tests is making progress 17:09:06 but we don't really have a defined end point 17:09:19 I think there are 4 or 5 patches in progress right now 17:09:58 I think we probably should just leave the bp open until release 17:10:12 just to make watching unit test reviews easier 17:10:33 then at summit I want to have some discussions about project policy on unit test requirements 17:10:41 sounds good 17:11:16 that's all the high priority blueprints, anything else we want to bring forward 17:11:44 well it'd be nice to get some eyes on my verify script bp 17:11:46 https://blueprints.launchpad.net/tempest/+spec/stop-leaking 17:11:51 I've got a few patches in progress 17:12:27 afazekas: how is the stop leaking bp progressing? 17:12:59 mtreinish: Started the new approach, which does not records the resources at the beginning of the run 17:13:15 https://review.openstack.org/#/c/78251/ currently it is failing 17:13:41 Two small nits related to this https://review.openstack.org/#/c/80280/ , https://review.openstack.org/#/c/78345/3 17:14:38 afazekas: cool, is there a review up yet? 17:14:51 So the new style is to try delete everything in the tenant before the tenent gets deleted 17:15:04 afazekas: +1 17:15:07 The tenent will be logged on creation to a database, or some kind of file 17:15:26 But the tenant deletion needs to be post process 17:15:53 afazekas: Why can't we delete stray resources when the tenant is getting deleted? 17:16:01 The process process will also double check is the tnent really empty (for example leak on setUpClass filures) 17:16:36 afazekas: so honestly, I'd like to fix the leaking problem by fixing the places where things leak 17:16:38 dkranz: the setUpclass failures are blind spots 17:16:40 not just cleanup at the end 17:16:41 afazekas: It would be better if you had a high-level description of the overall plan 17:16:48 afazekas: Rather than just a set of patches 17:16:54 afazekas: Is that possible? 17:17:30 on that front, what do people think about taking the nova approach for juno and having a blueprint gerrit repository 17:17:31 I will rewrite the BP, unless some would like to see the original plan implemented 17:17:53 afazekas: or start with a mailing list thread 17:18:02 because I feel like we still have some disconnect here 17:18:07 sdague: That is exactly what yair and I proposed for the neutron stuff . I still like it. 17:18:10 and I'd like to get us all on the same page 17:18:30 dkranz: which thing that I said? :) 17:18:43 sdague: gerrit blueprints 17:18:53 yeh 17:19:04 the setUpclasses could be enforced by a decorator, to call tearDownClass even on failuer, if is ok for everyone 17:19:09 sdague: yeah I think we can try that starting in juno 17:19:12 sdague: An important issue is whether we are managing resources as GC model or malloc/free 17:19:29 dkranz: I think it's got to be malloc / free 17:19:31 sdague: If the former than tenant cleanup is not to catch bugs but expected 17:19:52 sdague: I don't 100% disagree but why? 17:20:03 because otherwise we can run into races where we hit high water marks inside of tests 17:20:09 we've actually had those races before 17:20:29 sdague: But we already to gc at the class level which is the same as the tenant isolation level 17:20:33 do 17:20:54 tenant isolation is not going to work for lots of real environments 17:21:01 dkranz: With the new version the behavior on unexpected resource is configureable 17:21:12 afazekas: I see 17:21:26 and if we turn it into just GC, then people won't fix the issues in the tests 17:21:30 which are totally fixable 17:21:50 so I am -2 on automated cleanup at the end of tests 17:21:56 test classes / tenants 17:22:05 because I want us to be doing this right in the code 17:22:22 yeah we shouldn't auto cleanup just report the issues (or fail) if there is a leak 17:22:38 sdague: Are you ok with the clean + raise excpetion settings (default) 17:23:00 afazekas: for starting, I just want audit 17:23:17 and once we get clean, I'm ok with failing if we have leaked resources 17:23:44 but I don't want tempest doing failsafe brute force cleanup on class exit 17:23:53 sdague: I would be more comfortable if we did that 17:23:58 because we know that not all resources in openstack are actually discoverable 17:24:14 sdague: that is unfortunate and I would call it a bug 17:24:14 especially a bunch of the network ones 17:24:18 dkranz: sure 17:24:19 On successful run we do not leak mach 17:25:00 afazekas: so build an audit report at the end of classes some how and pull that together 17:25:06 that will help us fix things for real 17:25:07 sdague: Basically I think this is really important but we should decide the semantics up front 17:25:15 sure 17:25:22 afazekas: sounds good 17:25:29 so mailing list thread is fine at this point 17:25:45 ok is there anything else on blueprints? 17:25:52 otherwise let's move on to the next topic 17:25:59 Just logging is also configurable .. 17:26:22 #topic Neutron testing 17:26:44 mlavalle: are your around? 17:26:51 hi 17:27:01 any updates on neutron testing 17:27:10 Review of api tests have continued 17:27:29 I sent a message to the ML with 6 tests that were close to merge 17:27:51 They all were reviewed by cores over the past 3 days, thanks you :-) 17:28:06 2 of them merged and the other required chhanges 17:28:24 we have merged 12 api tests over the past 3 weeks 17:28:39 I have also identified 5 tests that are abandones 17:28:54 I am contacting the authors to see if they have still bandwidth 17:29:07 if not, will reassign tests to someone else 17:29:15 all in all, good progress 17:29:29 mlavalle: ok cool thanks 17:29:37 that's all I have 17:29:42 mlavalle: this is only slightly related to neutron testing, but I think it worth mentioning 17:29:49 will continue pushing api tests reviews 17:30:18 * mlavalle listening 17:30:24 mlavalle: we have a run_ssh flag in tempest, which is turned off by default, so a number of API tests are not doing ssh checks. But all new tests are ignoring it. 17:31:02 andreaf: You mean ignoring it and doing ssh checks anyway? 17:31:05 mlavalle: did you try turning that on in neutron devstack? I wonder if we should just remove the flag at all 17:31:28 andreaf: no, I haven't touched that flag 17:31:40 dkranz: yes 17:31:57 andreaf: after this https://review.openstack.org/#/c/54318/ , I will rebase this https://review.openstack.org/#/c/50337/ 17:31:58 dkranz: it must be as one of the main gate issue is ssh failures 17:32:15 andreaf: so I believe that was because in some environments tempest can't route to the guests 17:32:51 Well obviously we need to be consistent about this 17:33:02 honestly, I'd be ok with just removing the flag if it's getting ignored so much. How terrible are things if we try to default it true? 17:34:02 sdague: I'd be ok with removal too but defaulting to true does not address the issue 17:34:07 sdague: with nova networking we may need afakekas' patch to add floating IP to servers - but in a single node devstack not even that 17:34:25 dkranz: defaulting true first tells us how terrible the gate would collapse if we did that 17:34:40 it's more of a sniff to figure out if it's a truly terrible idea 17:34:47 sdague: the current ssh_check code expects fixed ip connection, and password injection, so it needs be able to use at least floating_ip with neutron 17:34:47 sdague: oh, that issue. I was talking about random tests failing if you set it to false if they are not checking 17:35:08 sdague: I tried that long time ago and it wasn't going too well, but the remote client has much improved since 17:35:27 afazekas: it won't handle cloud-init key injection? 17:35:45 ok, so honestly, I think this is probably a bigger issue than we want to bite of at this point in the release 17:36:00 sdague: I will rebase it https://review.openstack.org/#/c/54318/ soon, it is able to handle 17:36:10 but I think we should have a summit session on this to make sure we have a solid approach 17:36:12 What use is a cloud if a "user" (tempest) cannot access the created vms? 17:36:16 sdague: the key injection is working 17:36:30 sdague: Are suggesting we fix the tests that are ignoring the flag, or just ignore this issue for now? 17:36:31 but the password (file injection) is not 17:36:40 dkranz: I suggest ignoring the issue for now 17:36:48 but the cirros has default passwd as well 17:36:51 sdague: works for me 17:37:07 and once the release happens, we sort out a consistent approach for this 17:37:15 sdague: If some one fails due to not checking they may file a bug :) 17:37:21 sorry this one: https://review.openstack.org/#/c/50337/ 17:37:23 yes, sure 17:37:34 sdague: But that should have happened already it=f it was going to 17:38:06 yep, we can fix things as hot bugs right now, I just don't want to handle the whole problem, as we've got enough to worry about with the release 17:38:31 sdague: ok makes sense 17:38:49 ok, I think we can move on the next topic 17:38:49 it would be good to get this cleaned up early in juno for sure though 17:38:51 sdague: but was shall we do for new tests? 17:38:58 thanks andreaf for bringing it up 17:39:04 andreaf: use the run_ssh flag 17:39:07 as intended 17:39:19 #topic Heat testing 17:39:35 so I think we covered this one in the bp topic 17:39:41 sdague: unless you had something to add 17:39:52 nope, all covered 17:39:57 ok then let's move on 17:40:03 #topic Bugs 17:40:25 so I saw that maurosr sent an email out the ML about the bug day 17:40:58 I think he was proposing we have the bug day next Wed. 17:41:29 does anyone have any issues with that date? 17:41:34 nope, sounds good 17:41:57 ok cool so hopefully next week he'll have a summary of the bug day for the meeting 17:42:22 let's move on to the next topic then 17:42:41 oh unless someone wants to raise attention on a bug 17:42:59 https://review.openstack.org/#/c/77602/ 17:43:21 andreaf: heh well I guess thats a queue for the next topic 17:43:25 mtreinsh: nothing critical, it just need a +A then I can close the bug, very tiny review I promise 17:43:30 #topic Critical Reviews 17:43:38 mtreinish: ok sorry about that 17:43:38 #link https://review.openstack.org/#/c/77602/ 17:44:09 so does anyone else have any reviews they would like to bring up? 17:44:23 andreaf: lgtm 17:44:43 sdague: thanks 17:44:45 andreaf: I did find an earlier patch ended up dropping all the admin tests 17:44:53 so we lost 500 tests for a few days 17:45:10 so just as an fyi, be careful in checking test counts in the runs with auth related patches 17:45:25 this one looks right, still 2200 tests run in tempest-full 17:45:56 it does make me think about the idea of instituting low water mark checking 17:46:07 sdague: oh, thanks for letting me know, I'll be more careful 17:46:20 because we know that tempest full should run 2200 tests 17:46:27 so if it goes below 2000 17:46:32 something is wrong 17:46:55 sdague: sounds good, but you need to tune that based on your tempest.conf -> number of skips 17:47:10 andreaf: yeh, more like for the gate 17:47:30 I wouldn't implement it in tempest.conf, but in something else we call in the gate 17:48:07 anyway, this is a good patch, thanks for doing all this disconnect work 17:48:12 any other reviews? 17:48:35 ok if there aren't any other reviews let's move on to the last topic on the agenda 17:48:52 https://review.openstack.org/#/c/75411/ 17:49:14 #link https://review.openstack.org/#/c/75411/ 17:49:56 It fixes low chance random gate issue 17:50:04 afazekas: ok, I'll take a look at it after the meeting 17:50:12 unless someone beats me to it 17:50:29 ok let's move on 17:50:33 #topic Running tempest as non-admin (dkranz) 17:50:37 dkranz: you're up 17:50:50 afazekas: looks good 17:51:50 dkranz: ??? 17:52:39 ok well if dkranz isn't around does anyone have anything to say about this topic? 17:52:42 dkranz, mtreinish: while we wait for dkranz, I'd like to comment on this 17:52:57 andreaf: sure 17:53:04 mtreinish: as part of the multi-auth bp, I shall change tenant isolation to support v3 17:53:23 so it will be possible to create users and tenants within a domain 17:53:37 which only requires a domain admin, rather than an overall identity admin 17:53:53 sounds good to me 17:53:57 so with that in place it will be possible to get tenant isolation with a less powerful admin 17:54:07 but you still need identity admin to create the domain 17:54:16 oh nm I see what you're saying you specify a domain to run in 17:54:32 we can use the default domain which exists 17:54:36 andreaf: sounds very useful 17:55:13 I still think we should support a fallback of "specify users you want me to use" and limit running to that many processes 17:55:17 this is just one part of the story, there are many tests which rely on admin account for doing compute volume network stuff 17:55:17 The question is: is it enough for everyone who want to run tempest as non-admin ? 17:55:23 yeah that'll be nice to enable running in parallel without an admin 17:56:18 ok, so in the 5 minutes we have left, I want to just float the qa-specs gerrit repository idea for official 17:56:33 sdague: sure 17:56:45 #topic qa-specs gerrit repository 17:56:52 because I think we've had at least 3 different topics just in this meeting which should get a design for a blueprint 17:57:26 and I think there is no time like the present to try this new idea 17:57:35 I got booted out. 17:57:41 mtreinish: We don't have a way to run tempest as non-admin and skip all admin tests, right? 17:57:56 so my suggestion is I get the repository set up 17:58:18 and we start doing these sorts of things there in review format 17:58:36 dkranz: don't specify an admin in the config and the tests should get skipped 17:58:39 dkranz: that's not true, because in one of andreaf's refactorings the admin tenant was lost 17:58:40 if they don't it's a bug 17:58:47 and we skipped 500 tests 17:58:53 sdague: yeah that's sounds fine to me 17:58:59 mtreinish: ok, cool. I didn't know that. 17:59:44 #info qa-specs repo to be created for blueprint design / discussion / approval 17:59:51 sdague: well we're out of time... 18:00:03 thanks everyone 18:00:05 #endmeeting