17:01:01 #startmeeting qa 17:01:02 Meeting started Thu Feb 27 17:01:01 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:03 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:05 The meeting name has been set to 'qa' 17:01:08 hi who do we have today? 17:01:30 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 17:01:44 ^^^ Today's agenda (looks like the standard boiler plate) 17:01:47 mtreinish: Here 17:01:52 hi 17:02:01 o/ 17:02:45 sdague, mkoderer, afazekas: are you guys around today? 17:02:54 yes 17:03:10 ok well let's get started 17:03:10 yes 17:03:15 hi 17:03:22 #topic Blueprints 17:03:38 so does anyone have any blueprints they would like to give a status update on 17:03:52 or any new bp proposals to discuss? 17:03:54 mtreinish: ok I'll go first 17:04:07 we should go through everything in high 17:04:12 andreaf: go for it 17:04:26 #link https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests 17:04:48 and more importantly #link https://blueprints.launchpad.net/tempest/ 17:06:17 I'm back 17:06:27 '' mtreinish: ok I'll go first 17:06:43 ' andreaf: go for it' 17:06:48 mtreinish: I was saying the next step for that bp is available for review #link https://review.openstack.org/#/c/74387/ 17:07:22 andreaf_: ok, so after that the next step is to convert cred usage to the new object 17:07:27 where do you go from there? 17:07:30 mtreinish: so it's progressing ok but slowly it's a lot of small pieces so reviews would be very welcome 17:08:01 mtreinish: support for v3 for official client and scenario tests 17:08:17 mtreinish: support for v3 api (not token, also user create) in tenant isolation 17:08:31 andreaf_: heh, well client support might take some time 17:08:35 mtreinish: support for v3 in stress, cli, third party 17:09:12 mtreinish: it's possible to create several of the clients using token and baseurl 17:09:38 mtreinish: a colleague of mine just got merged a change in novaclient for ostoken 17:10:04 mtreinish: so hopefully it's not too bad - in the meantime I can also work on the other bits 17:10:16 andreaf_: ok, cool 17:10:35 andreaf_: is there anything else for multi-auth? 17:10:47 mtreinish: another things which I wanted to do actually is complete refactor of manager class - to provide lazy load of clients 17:11:20 mtreinish: not striclty needed for this bp but useful to simplify a few things 17:11:32 mtreinish: that's all 17:11:41 it would make things 20ns faster :) 17:12:00 andreaf_: ok that sounds like an interesting idea, so that way we wouldn't be doing the initial handshake on imports anymore right? 17:12:19 mtreinish: right 17:12:28 well I definitely support that :) 17:12:55 ok lets move onto another high prio bp 17:12:57 #link https://blueprints.launchpad.net/tempest/+spec/unit-tests 17:13:04 so this one is owned by me 17:13:14 how's it looking for i3? 17:13:20 from last week we've had a few more unit test patches mostly related to auth stuff 17:13:38 I need to make a push to get some more coverage for some of the common functionality 17:13:45 but we've been making progress 17:13:52 sdague: it might be close 17:14:00 but we should have good coverage for the release 17:14:12 ok, great 17:14:14 I also added the coverage job to the post runs for the tempest queue 17:14:20 very good 17:14:38 I also want to discuss making a project policy change about requiring unit tests for new functionality 17:14:45 but that may be a summit discussion 17:14:48 mtreinish: https://review.openstack.org/#/c/74387/ which I mentioned before also include unit tests for the new credentials class 17:15:15 mtreinish: +1 it's really useful 17:15:25 agreed 17:15:32 it's a nice new culture in the codebase 17:15:40 most of tempest code is tested on the gate , at least the success code path anyway, we might need to focus on the fail path in unit tests 17:15:56 ok, moving on: #link https://blueprints.launchpad.net/tempest/+spec/negative-tests 17:16:06 dkranz: comments on that? 17:16:23 also afazekas #link https://blueprints.launchpad.net/tempest/+spec/stop-leaking 17:16:28 I am still trying to get time to push another example for a different service. 17:16:31 that's been listed as high for a long time 17:16:44 sdague: It would be nice to get https://review.openstack.org/#/c/73982/ approved. 17:16:47 we should decided if it's dead, or when it's going to get done 17:17:09 sdague: After that is approved I was going to send a message to the list inviting people to contribute negative auto-gen tests. 17:17:23 dkranz: can you update the status on the blueprint to something other than none? 17:17:49 sdague: Yes, didn't realize it was still there 17:17:49 dkranz: ok, I'll review that by the end of the week. 17:17:56 sdague: Since the first global leak think is not liked, I am planning to do leak detection (and also cleanup) stuff at the isolated tenant deletion 17:18:18 sdague: I'm not sure how to say when this blueprint is "done". All the infrastructure is there. 17:18:30 sdague: It's now a matter of converting existing negative tests 17:18:35 dkranz: I'd say then break it up 17:18:40 blueprint for infrastructure 17:18:46 which we can declare done 17:18:58 then blueprint for some set of test conversion (or a bug) 17:19:10 sdague: or a spreadsheet... 17:19:10 dkranz: but there is already another bp mentioned for that "moving the current negative tests into separate files: https://blueprints.launchpad.net/tempest/+spec/negative-test-files" 17:19:13 afazekas: ok, is that something that's currently in progress 17:19:16 sdague: ok, I'll close the current as infrastructure and create another one 17:19:19 afazekas: what about if isolation is disabled? 17:19:20 mtreinish: or spreadsheet 17:19:30 dkranz: oh sorry, ignore me 17:19:33 psedlak: That is a slightly different thing 17:19:49 psedlak: Mostly done I think 17:19:50 mtreinish: well, at least in tenant isolation we should be able to easily leak detect on tenant cleanup 17:20:00 and fix it in the main code 17:20:15 sdague: recently I analyses some flaky things, but I hope I will have time on working on that soon 17:20:17 we shouldn't clean up in tenant cleanup, just use it as a detector 17:20:45 afazekas: ok, well we need timeline commitments if it's going to be high, otherwise I'll bump it down and push to juno 17:20:54 which is basically the question 17:21:43 sdague: I think on next week I will check some flaky stuff too, but after that I will have time 17:21:44 sdague: what about the cleanup in isolation code being optional? (to keep the possibility of getting clean env after tempest) 17:22:06 psedlak: no, I want to just detect, then go fix the tests 17:22:18 otherwise we fail on the non isolated case 17:22:33 psedlak: the credentials and tenants created for isolation will still be removed 17:22:36 sdague: just for detecting my first thing was good enough.. 17:22:42 sdague: sure, but why not to have at least capability to clean in there? 17:22:55 mtreinish: but not the leaked servers etc ... 17:23:04 psedlak: then that's a real bug that needs to be fixed 17:23:08 psedlak: because people will stop trying to fix the real leaks 17:23:17 this is social engineering as much as anything else 17:23:18 ok 17:23:53 ok what else is on the high list? 17:23:56 ok, I bumped to low and removed the target. Once it's a high focus again we can bump it up 17:24:01 neutron-full and heat 17:24:11 and nova v3 17:24:17 oh oops I forgot to add the heat topic this week 17:24:25 #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests 17:24:44 well it's a bit late for cyeoh right now 17:24:47 I just removed the target, I think all the nova api discussion means we have to figure out the outcome there 17:24:56 sdague: yeah I agree 17:25:05 #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full 17:25:16 rossella_s: you around? 17:25:34 I should probably change the blueprint name there :) 17:25:43 heh 17:25:53 or stevebaker on heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration 17:27:08 I guess not 17:27:11 sdague: well I think we should just move on maybe ask about those 2 on the ML 17:27:13 or we are netsplit 17:27:25 sdague: I don't think so 17:27:26 yeh, you want to take that? or should I 17:27:36 sdague: it seems more PTL like :) 17:27:50 #action sdague to ask about current high priority blueprints on list 17:28:05 ok lets move on to the next topic 17:28:14 #topic Neutron testing 17:28:33 so I don't see mlavalle on right now 17:28:38 salv-orlando: are you around? 17:28:47 #link https://review.openstack.org/#/c/66375/ 17:28:51 yeah I am 17:28:54 sdague: we have a non-voting full parallel job now right? 17:29:06 salv-orlando: anything to share about neutron testing? 17:29:12 The above change can fix one of the most frequent ovs agent issue 17:29:22 I think there are a bunch of patches for neutron API testing under review 17:29:48 and we have assignees working on the most pressing issues to get the full tempest job work in a way such that we can make it voting 17:30:26 There are also a bunch of patches from yfried for scenario testing. 17:30:37 I am struggling to find time to give them a proper review however. 17:30:43 salv-orlando: ok cool 17:30:52 salv-orlando: yeah I've been a bit lax with reviews myself lately 17:30:52 yes 17:30:53 that's all from neutron 17:31:04 salv-orlando: awesome, thanks 17:31:25 ok then I guess let's move on to the next topic 17:31:32 #topic Bugs 17:31:33 salv-orlando: nice work. 17:31:52 so I haven't had a chance to look at the bug list since last week 17:32:07 I definitely think we're going to need to do another bug day before release 17:32:16 does anyone want to volunteer to take the lead on it? 17:32:18 yeh, I'd wait until post i3 17:32:21 salv-orlando: What do you think, is it help full in debuging neutron issues on the gate https://review.openstack.org/#/c/76643/ 17:32:29 anyone volunteering for organizing one? 17:33:03 crickets :) 17:33:21 ok, mtreinish want to stick find volunteer for bug day on next agenda? 17:33:27 sdague: sure will do 17:33:35 afazekas: all these information *might* be helpful even if I've personally not used anything from ip_log_ns in the past month, ovs_db info are definitely more useful 17:33:58 ok time for the last topic on the agenda 17:34:13 #topic Critical Reviews 17:34:26 does anyone have any reviews that they would like to bring up to get some eyes on? 17:34:52 I have one which goes into the land of open discussion, not quite critical 17:35:01 mtreinish: I added one more topic "What is our strategy for getting fail on log errors turned back on?" 17:35:09 I resurected the parallel filter - https://review.openstack.org/#/c/76674/ 17:35:16 https://review.openstack.org/#/c/75411/ small change against az flaky 17:35:28 dkranz: I see that now, did you just add it? 17:35:37 sdague: yeah I'm planning to take a look at that today 17:35:38 mtreinish: Right at the start of the meeting 17:35:43 as salv-orlando mentioned the yfried's chain ... 17:35:46 curious on output comments, I could also mark the worker thread and do colorizing on the os-loganalyze side 17:35:46 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z 17:35:59 though I'm going to work through a nova bug today, so that won't be until next week 17:36:00 sdague: but I like that you add configurable length to the indent 17:36:04 #link https://review.openstack.org/#/c/76674/ 17:36:05 mtreinish: that's in there now 17:36:16 #link https://review.openstack.org/#/c/75411/ 17:36:26 sdague: yeah I know I was saying it was good :) 17:36:27 L137 - https://review.openstack.org/#/c/76674/2/tools/subunit-trace.py 17:36:42 sdague: the current output looks good, colorizing could be even nicer 17:37:03 yeh, I agree that I'm mixed on it when we get to 4 processes 17:37:03 andreaf_: we have the colorizer in tools/ already 17:37:21 andreaf_: or did you meant use color smarter in the new filter? 17:37:24 yeh, the colorizer will need to be different, because this hooks the subunit stream at a different place 17:38:06 mtreinish: yes I though use different color for different processes 17:38:28 mtreinish: at the moment indentation is used 17:38:38 andreaf_: yeh, that's what I was thinking, but from a web perspective, we'll have to do that in os-loganalyze 17:39:04 sdague: but this comes back to what I was saying about adding options for local runs to the filter too 17:39:08 anyway, will keep puttering on it, comments welcome 17:39:10 mtreinish: sure 17:39:26 ok are there any other reviews? 17:39:41 afazekas: on https://review.openstack.org/#/c/75411/3 why not deal with that with a lock? 17:39:54 that's who we handle az manip in osapi 17:40:08 sdague: it is different type of issue 17:40:36 ok, will look later 17:40:38 Long time ago we just had only one AZ one the gate, and the test pick the first existing az 17:40:53 Now other test cases are creating and deleting az 17:40:54 right, that's definitely not right :) 17:41:16 afazekas: why not just create your own boto test az then? 17:41:27 if a lock won't fix the issue 17:41:51 well anyway afazekas we can talk about this on -qa later 17:41:51 mtreinish: one az exists any all system anyway 17:42:19 lets move on to dkranz's topic 17:42:24 yep 17:42:36 #topic What is our strategy for getting fail on log errors turned back on? (dkranz) 17:42:38 sdague and I discussed this a little bit yesterday 17:42:40 dkranz: ok go ahead 17:42:56 I did a lot of work putting together the whitelist as it is now 17:43:02 I am not eager to do that again 17:43:22 So the question is how we can get a clean run 17:43:44 Seems to me we should approach this one log file at a time 17:44:07 For each log file we get the project to clean it, then we start failing that file 17:44:15 And do on until we hit them all 17:44:27 * afazekas thinks seeing just real ERRORs in the log has high value 17:44:29 so 17:44:41 afazekas: real ERRORs should be failures 17:44:56 sdague: yes 17:44:56 if the system has an error, it shouldn't pass the tests 17:45:00 afazekas: I agree and we had that happening but turned it off due to flaky gate 17:45:03 dkranz: that might work, then we can go about it more iteratively instead of trying to do it all at once 17:45:11 mtreinish: exactly 17:45:41 dkranz: ok, so marking logs that aren't allowed to have errors? 17:45:56 so there needs to be some support on that in your tool 17:45:57 sdague: right. It should be a trivial change to the checker 17:46:03 sdague: I will do that 17:46:04 I'd be ok with that 17:46:17 sdague: Can you socialize this at the project meeting? 17:46:23 sure 17:46:25 sdague: We need some volunteers 17:46:29 sdague: to go first 17:46:37 well, volunteers right now will be hard 17:46:41 with the rush on i3 17:46:58 sdague: I know. But it we start talking about it now maybe after that we can get some progress 17:47:02 sure 17:47:10 dkranz: on a related topic, but on slightly, does the log check tool support pulling logs from a multinode test environment? 17:47:58 andreaf_: Not really. Unless you use central logging 17:47:59 andreaf_: you can cat it before processing :) 17:48:22 dkranz, afazekas: ok 17:48:35 anderstj: It just takes a directory where it expects to find the logs 17:48:45 andreaf_: or url 17:48:57 anderstj: Sorry 17:49:16 mtreinish: I think that's all for that topic 17:49:23 dkranz: ok thanks 17:49:31 dkranz: ok I see 17:49:35 well that leads us to the open discussion section then 17:49:39 #topic open discussion 17:49:56 so does anyone have any topics that they'd like to bring up that weren't on the agenda? 17:49:59 Periodic master jobs are not in good state 17:50:07 mtreinish: I'm starting to look in tripleo-ci, I'd like to get tempest running on the overcloud in there 17:50:17 dkranz: what about picking first volunteer(s) based on the amount of errors ... start with those which have just few 17:50:23 afazekas: sorry go ahead 17:50:30 afazekas: I just did a buch of changes to what was running in the periodic jobs 17:50:42 psedlak: Sure, except that we can't pick them. They have to volunteer :) 17:50:43 because the list was really messed up before 17:50:56 We have some peridic jobs (including stress), but according to the log they are failed to start :( 17:50:56 so I think we may have had some bit rot because they weren't running 17:51:29 afazekas: yeah I noticed that too, I'll have to dig more into it 17:51:33 dkranz: sure, just we can try to convince them to volunteer a with less errors the resistance could be smaller ;) 17:51:50 andreaf_: so what is needed to get tempest running there? 17:52:13 dkranz: anyway if any project will agree in the first round we don't need to care about picking one 17:52:14 mtreinish: well first of all get tempest deployed and then configure it 17:52:18 mtreinish: at the first look the job definition was the same as with the working jobs 17:52:53 mtreinish: I just started looking into it, I wanted to know if anyone is already on it from this team 17:53:24 * afazekas thinking about some additional stress jobs with the know flaky issues 17:53:37 andreaf_: I haven't been and it hasn't come up on the meeting before. But, I'm sure someone working the tripleo ci stuff is looking at it 17:53:47 because as I understood it the plan was to run tempest for that 17:54:03 mtreinish: as tripleo supports multi node environments potentially, there may be some things to be changed to make everything work in multinode scenario as well 17:54:07 afazekas: can't you just add the decorator the flaky tests? 17:54:34 andreaf_: Yes, part of the work here is to de-devstackize tempest configuration 17:54:43 andreaf_: Which I have also been looking at. 17:55:11 dkranz: great 17:55:24 mtreinish: I would like to have ~4 jobs with only one or two test on 4-8 threads 17:56:07 afazekas: it's been a while since I looked at the stress stuff but I think that should be doable with the current framework 17:56:09 mtreinish: do we have bps on tempest side for this topics (de-devstackize, multinode) 17:56:14 mtreinish: If you have zillion of different things on multiple thread it is difficult to tell what really contributed to the flaky result 17:56:37 andreaf_: well there was a multinode one for like forever 17:56:42 mtreinish: yes it is 17:57:01 andreaf_: as for de-devstack stuff I don't think so 17:57:03 andreaf_: https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator 17:57:10 #link https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator 17:57:20 but it's something I always look out for in reviews 17:57:48 #link https://bugs.launchpad.net/nova/+bug/1284708 17:58:42 psedlak: its more about underlying assumptions in tempest about the env 17:58:44 psedlak: thanks. For tripleo-ci, as we have the config data which is used to setup the environment (like for devstack), I was thinking we could configure tempest based on it - same way we configure services via heat in the overcloud 17:58:56 mtreinish: Right 17:59:29 mtreinish: Like I just discovered tempest expects the admin user to also have admin role in the demo tenant 18:00:07 andreaf_: Yes you could do that. 18:00:08 ok well we're out of time for today 18:00:15 thanks everyone 18:00:19 bye 18:00:20 #endmeeting