17:01:01 <mtreinish> #startmeeting qa
17:01:02 <openstack> Meeting started Thu Feb 27 17:01:01 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:05 <openstack> The meeting name has been set to 'qa'
17:01:08 <mtreinish> hi who do we have today?
17:01:30 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:44 <mtreinish> ^^^ Today's agenda (looks like the standard boiler plate)
17:01:47 <dkranz> mtreinish: Here
17:01:52 <andreaf> hi
17:02:01 <jaypipes> o/
17:02:45 <mtreinish> sdague, mkoderer, afazekas: are you guys around today?
17:02:54 <sdague> yes
17:03:10 <mtreinish> ok well let's get started
17:03:10 <afazekas> yes
17:03:15 <psedlak> hi
17:03:22 <mtreinish> #topic Blueprints
17:03:38 <mtreinish> so does anyone have any blueprints they would like to give a status update on
17:03:52 <mtreinish> or any new bp proposals to discuss?
17:03:54 <andreaf> mtreinish: ok I'll go first
17:04:07 <sdague> we should go through everything in high
17:04:12 <sdague> andreaf: go for it
17:04:26 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests
17:04:48 <sdague> and more importantly #link https://blueprints.launchpad.net/tempest/
17:06:17 <andreaf_> I'm back
17:06:27 <afazekas> ''<andreaf> mtreinish: ok I'll go first
17:06:43 <afazekas> '<sdague> andreaf: go for it'
17:06:48 <andreaf_> mtreinish: I was saying the next step for that bp is available for review #link https://review.openstack.org/#/c/74387/
17:07:22 <mtreinish> andreaf_: ok, so after that the next step is to convert cred usage to the new object
17:07:27 <mtreinish> where do you go from there?
17:07:30 <andreaf_> mtreinish: so it's progressing ok but slowly it's a lot of small pieces so reviews would be very welcome
17:08:01 <andreaf_> mtreinish: support for v3 for official client and scenario tests
17:08:17 <andreaf_> mtreinish: support for v3 api (not token, also user create) in tenant isolation
17:08:31 <mtreinish> andreaf_: heh, well client support might take some time
17:08:35 <andreaf_> mtreinish: support for v3 in stress, cli, third party
17:09:12 <andreaf_> mtreinish: it's possible to create several of the clients using token and baseurl
17:09:38 <andreaf_> mtreinish: a colleague of mine just got merged a change in novaclient for ostoken
17:10:04 <andreaf_> mtreinish: so hopefully it's not too bad - in the meantime I can also work on the other bits
17:10:16 <mtreinish> andreaf_: ok, cool
17:10:35 <mtreinish> andreaf_: is there anything else for multi-auth?
17:10:47 <andreaf_> mtreinish: another things which I wanted to do actually is complete refactor of manager class - to provide lazy load of clients
17:11:20 <andreaf_> mtreinish: not striclty needed for this bp but useful to simplify a few things
17:11:32 <andreaf_> mtreinish: that's all
17:11:41 <afazekas> it would make things 20ns faster :)
17:12:00 <mtreinish> andreaf_: ok that sounds like an interesting idea, so that way we wouldn't be doing the initial handshake on imports anymore right?
17:12:19 <andreaf_> mtreinish: right
17:12:28 <mtreinish> well I definitely support that :)
17:12:55 <mtreinish> ok lets move onto another high prio bp
17:12:57 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/unit-tests
17:13:04 <mtreinish> so this one is owned by me
17:13:14 <sdague> how's it looking for i3?
17:13:20 <mtreinish> from last week we've had a few more unit test patches mostly related to auth stuff
17:13:38 <mtreinish> I need to make a push to get some more coverage for some of the common functionality
17:13:45 <mtreinish> but we've been making progress
17:13:52 <mtreinish> sdague: it might be close
17:14:00 <mtreinish> but we should have good coverage for the release
17:14:12 <sdague> ok, great
17:14:14 <mtreinish> I also added the coverage job to the post runs for the tempest queue
17:14:20 <sdague> very good
17:14:38 <mtreinish> I also want to discuss making a project policy change about requiring unit tests for new functionality
17:14:45 <mtreinish> but that may be a summit discussion
17:14:48 <andreaf_> mtreinish: https://review.openstack.org/#/c/74387/ which I mentioned before also include unit tests for the new credentials class
17:15:15 <andreaf_> mtreinish: +1 it's really useful
17:15:25 <sdague> agreed
17:15:32 <sdague> it's a nice new culture in the codebase
17:15:40 <afazekas> most of tempest code is tested on the gate , at least the success code path anyway, we might need to focus on the fail path in unit tests
17:15:56 <sdague> ok, moving on: #link https://blueprints.launchpad.net/tempest/+spec/negative-tests
17:16:06 <sdague> dkranz: comments on that?
17:16:23 <sdague> also afazekas #link https://blueprints.launchpad.net/tempest/+spec/stop-leaking
17:16:28 <dkranz> I am still trying to get time to push another example for a different service.
17:16:31 <sdague> that's been listed as high for a long time
17:16:44 <dkranz> sdague: It would be nice to get https://review.openstack.org/#/c/73982/ approved.
17:16:47 <sdague> we should decided if it's dead, or when it's going to get done
17:17:09 <dkranz> sdague: After that is approved I was going to send a message to the list inviting people to contribute negative auto-gen tests.
17:17:23 <sdague> dkranz: can you update the status on the blueprint to something other than none?
17:17:49 <dkranz> sdague: Yes, didn't realize it was still there
17:17:49 <sdague> dkranz: ok, I'll review that by the end of the week.
17:17:56 <afazekas> sdague: Since the first global leak think is not liked, I am planning to do leak detection (and also cleanup) stuff at the isolated tenant deletion
17:18:18 <dkranz> sdague: I'm not sure how to say when this blueprint is "done". All the infrastructure is there.
17:18:30 <dkranz> sdague: It's now a matter of converting existing negative tests
17:18:35 <sdague> dkranz: I'd say then break it up
17:18:40 <sdague> blueprint for infrastructure
17:18:46 <sdague> which we can declare done
17:18:58 <sdague> then blueprint for some set of test conversion (or a bug)
17:19:10 <mtreinish> sdague: or a spreadsheet...
17:19:10 <psedlak> dkranz: but there is already another bp mentioned for that "moving the current negative tests into separate files: https://blueprints.launchpad.net/tempest/+spec/negative-test-files"
17:19:13 <sdague> afazekas: ok, is that something that's currently in progress
17:19:16 <dkranz> sdague: ok, I'll close the current as infrastructure and create another one
17:19:19 <mtreinish> afazekas: what about if isolation is disabled?
17:19:20 <sdague> mtreinish: or spreadsheet
17:19:30 <psedlak> dkranz: oh sorry, ignore me
17:19:33 <dkranz> psedlak: That is a slightly different thing
17:19:49 <dkranz> psedlak: Mostly done I think
17:19:50 <sdague> mtreinish: well, at least in tenant isolation we should be able to easily leak detect on tenant cleanup
17:20:00 <sdague> and fix it in the main code
17:20:15 <afazekas> sdague:  recently I analyses some flaky things, but I hope I will have time on working on that soon
17:20:17 <sdague> we shouldn't clean up in tenant cleanup, just use it as a detector
17:20:45 <sdague> afazekas: ok, well we need timeline commitments if it's going to be high, otherwise I'll bump it down and push to juno
17:20:54 <sdague> which is basically the question
17:21:43 <afazekas> sdague: I think on next week I will check some flaky stuff too, but after that I will have time
17:21:44 <psedlak> sdague: what about the cleanup in isolation code being optional? (to keep the possibility of getting clean env after tempest)
17:22:06 <sdague> psedlak: no, I want to just detect, then go fix the tests
17:22:18 <sdague> otherwise we fail on the non isolated case
17:22:33 <mtreinish> psedlak: the credentials and tenants created for isolation will still be removed
17:22:36 <afazekas> sdague: just for detecting my first thing was good enough..
17:22:42 <psedlak> sdague: sure, but why not to have at least capability to clean in there?
17:22:55 <psedlak> mtreinish: but not the leaked servers etc ...
17:23:04 <mtreinish> psedlak: then that's a real bug that needs to be fixed
17:23:08 <sdague> psedlak: because people will stop trying to fix the real leaks
17:23:17 <sdague> this is social engineering as much as anything else
17:23:18 <psedlak> ok
17:23:53 <mtreinish> ok what else is on the high list?
17:23:56 <sdague> ok, I bumped to low and removed the target. Once it's a high focus again we can bump it up
17:24:01 <sdague> neutron-full and heat
17:24:11 <sdague> and nova v3
17:24:17 <mtreinish> oh oops I forgot to add the heat topic this week
17:24:25 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
17:24:44 <mtreinish> well it's a bit late for cyeoh right now
17:24:47 <sdague> I just removed the target, I think all the nova api discussion means we have to figure out the outcome there
17:24:56 <mtreinish> sdague: yeah I agree
17:25:05 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
17:25:16 <sdague> rossella_s: you around?
17:25:34 <sdague> I should probably change the blueprint name there :)
17:25:43 <mtreinish> heh
17:25:53 <sdague> or stevebaker on heat - https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration
17:27:08 <sdague> I guess not
17:27:11 <mtreinish> sdague: well I think we should just move on maybe ask about those 2 on the ML
17:27:13 <sdague> or we are netsplit
17:27:25 <mtreinish> sdague: I don't think so
17:27:26 <sdague> yeh, you want to take that? or should I
17:27:36 <mtreinish> sdague: it seems more PTL like :)
17:27:50 <sdague> #action sdague to ask about current high priority blueprints on list
17:28:05 <mtreinish> ok lets move on to the next topic
17:28:14 <mtreinish> #topic Neutron testing
17:28:33 <mtreinish> so I don't see mlavalle on right now
17:28:38 <mtreinish> salv-orlando: are you around?
17:28:47 <afazekas> #link https://review.openstack.org/#/c/66375/
17:28:51 <salv-orlando> yeah  I am
17:28:54 <mtreinish> sdague: we have a non-voting full parallel job now right?
17:29:06 <mtreinish> salv-orlando: anything to share about neutron testing?
17:29:12 <afazekas> The above change can fix one of the most frequent ovs agent issue
17:29:22 <salv-orlando> I think there are a bunch of patches for neutron API testing under review
17:29:48 <salv-orlando> and we have assignees working on the most pressing issues to get the full tempest job work in a way such that we can make it voting
17:30:26 <salv-orlando> There are also a bunch of patches from yfried for scenario testing.
17:30:37 <salv-orlando> I am struggling to find time to give them a proper review however.
17:30:43 <mtreinish> salv-orlando: ok cool
17:30:52 <mtreinish> salv-orlando: yeah I've been a bit lax with reviews myself lately
17:30:52 <sdague> yes
17:30:53 <salv-orlando> that's all from neutron
17:31:04 <mtreinish> salv-orlando: awesome, thanks
17:31:25 <mtreinish> ok then I guess let's move on to the next topic
17:31:32 <mtreinish> #topic Bugs
17:31:33 <sdague> salv-orlando: nice work.
17:31:52 <mtreinish> so I haven't had a chance to look at the bug list since last week
17:32:07 <mtreinish> I definitely think we're going to need to do another bug day before release
17:32:16 <mtreinish> does anyone want to volunteer to take the lead on it?
17:32:18 <sdague> yeh, I'd wait until post i3
17:32:21 <afazekas> salv-orlando: What do you think, is it help full in debuging neutron issues on the gate https://review.openstack.org/#/c/76643/
17:32:29 <sdague> anyone volunteering for organizing one?
17:33:03 <sdague> crickets :)
17:33:21 <sdague> ok, mtreinish want to stick find volunteer for bug day on next agenda?
17:33:27 <mtreinish> sdague: sure will do
17:33:35 <salv-orlando> afazekas: all these information *might* be helpful even if I've personally not used anything from ip_log_ns in the past month, ovs_db info are definitely more useful
17:33:58 <mtreinish> ok time for the last topic on the agenda
17:34:13 <mtreinish> #topic Critical Reviews
17:34:26 <mtreinish> does anyone have any reviews that they would like to bring up to get some eyes on?
17:34:52 <sdague> I have one which goes into the land of open discussion, not quite critical
17:35:01 <dkranz> mtreinish: I added one more topic "What is our strategy for getting fail on log errors turned back on?"
17:35:09 <sdague> I resurected the parallel filter - https://review.openstack.org/#/c/76674/
17:35:16 <afazekas> https://review.openstack.org/#/c/75411/ small change against az flaky
17:35:28 <mtreinish> dkranz: I see that now, did you just add it?
17:35:37 <mtreinish> sdague: yeah I'm planning to take a look at that today
17:35:38 <dkranz> mtreinish: Right at the start of the meeting
17:35:43 <psedlak> as salv-orlando mentioned the yfried's chain ...
17:35:46 <sdague> curious on output comments, I could also mark the worker thread and do colorizing on the os-loganalyze side
17:35:46 <psedlak> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z
17:35:59 <sdague> though I'm going to work through a nova bug today, so that won't be until next week
17:36:00 <mtreinish> sdague: but I like that you add configurable length to the indent
17:36:04 <mtreinish> #link https://review.openstack.org/#/c/76674/
17:36:05 <sdague> mtreinish: that's in there now
17:36:16 <mtreinish> #link https://review.openstack.org/#/c/75411/
17:36:26 <mtreinish> sdague: yeah I know I was saying it was good :)
17:36:27 <sdague> L137 - https://review.openstack.org/#/c/76674/2/tools/subunit-trace.py
17:36:42 <andreaf_> sdague: the current output looks good, colorizing could be even nicer
17:37:03 <sdague> yeh, I agree that I'm mixed on it when we get to 4 processes
17:37:03 <mtreinish> andreaf_: we have the colorizer in tools/ already
17:37:21 <mtreinish> andreaf_: or did you meant use color smarter in the new filter?
17:37:24 <sdague> yeh, the colorizer will need to be different, because this hooks the subunit stream at a different place
17:38:06 <andreaf_> mtreinish: yes I though use different color for different processes
17:38:28 <andreaf_> mtreinish: at the moment indentation is used
17:38:38 <sdague> andreaf_: yeh, that's what I was thinking, but from a web perspective, we'll have to do that in os-loganalyze
17:39:04 <mtreinish> sdague: but this comes back to what I was saying about adding options for local runs to the filter too
17:39:08 <sdague> anyway, will keep puttering on it, comments welcome
17:39:10 <sdague> mtreinish: sure
17:39:26 <mtreinish> ok are there any other reviews?
17:39:41 <sdague> afazekas: on https://review.openstack.org/#/c/75411/3 why not deal with that with a lock?
17:39:54 <sdague> that's who we handle az manip in osapi
17:40:08 <afazekas> sdague: it is different type of issue
17:40:36 <sdague> ok, will look later
17:40:38 <afazekas> Long time ago we just had only one AZ one the gate, and the test pick the first existing az
17:40:53 <afazekas> Now other test cases are creating and deleting az
17:40:54 <sdague> right, that's definitely not right :)
17:41:16 <mtreinish> afazekas: why not just create your own boto test az then?
17:41:27 <mtreinish> if a lock won't fix the issue
17:41:51 <mtreinish> well anyway afazekas we can talk about this on -qa later
17:41:51 <afazekas> mtreinish: one az exists any all system anyway
17:42:19 <mtreinish> lets move on to dkranz's topic
17:42:24 <sdague> yep
17:42:36 <mtreinish> #topic What is our strategy for getting fail on log errors turned back on? (dkranz)
17:42:38 <dkranz> sdague and I discussed this a little bit yesterday
17:42:40 <mtreinish> dkranz: ok go ahead
17:42:56 <dkranz> I did a lot of work putting together the whitelist as it is now
17:43:02 <dkranz> I am not eager to do that again
17:43:22 <dkranz> So the question is how we can get a clean run
17:43:44 <dkranz> Seems to me we should approach this one log file at a time
17:44:07 <dkranz> For each log file we get the project to clean it, then we start failing that file
17:44:15 <dkranz> And do on until we hit them all
17:44:27 * afazekas thinks seeing just real ERRORs in the log has high value
17:44:29 <dkranz> so
17:44:41 <sdague> afazekas: real ERRORs should be failures
17:44:56 <afazekas> sdague: yes
17:44:56 <sdague> if the system has an error, it shouldn't pass the tests
17:45:00 <dkranz> afazekas: I agree and we had that happening but turned it off due to flaky gate
17:45:03 <mtreinish> dkranz: that might work, then we can go about it more iteratively instead of trying to do it all at once
17:45:11 <dkranz> mtreinish: exactly
17:45:41 <sdague> dkranz: ok, so marking logs that aren't allowed to have errors?
17:45:56 <sdague> so there needs to be some support on that in your tool
17:45:57 <dkranz> sdague: right. It should be a trivial change to the checker
17:46:03 <dkranz> sdague: I will do that
17:46:04 <sdague> I'd be ok with that
17:46:17 <dkranz> sdague: Can you socialize this at the project meeting?
17:46:23 <sdague> sure
17:46:25 <dkranz> sdague: We need some volunteers
17:46:29 <dkranz> sdague: to go first
17:46:37 <sdague> well, volunteers right now will be hard
17:46:41 <sdague> with the rush on i3
17:46:58 <dkranz> sdague: I know. But it we start talking about it now maybe after that we can get some progress
17:47:02 <sdague> sure
17:47:10 <andreaf_> dkranz: on a related topic, but on slightly, does the log check tool support pulling logs from a multinode test environment?
17:47:58 <dkranz> andreaf_: Not really. Unless you use central logging
17:47:59 <afazekas> andreaf_: you can cat it before processing :)
17:48:22 <andreaf_> dkranz, afazekas: ok
17:48:35 <dkranz> anderstj: It just takes a directory where it expects to find the logs
17:48:45 <dkranz> andreaf_: or url
17:48:57 <dkranz> anderstj: Sorry
17:49:16 <dkranz> mtreinish: I think that's all for that topic
17:49:23 <mtreinish> dkranz: ok thanks
17:49:31 <andreaf_> dkranz: ok I see
17:49:35 <mtreinish> well that leads us to the open discussion section then
17:49:39 <mtreinish> #topic open discussion
17:49:56 <mtreinish> so does anyone have any topics that they'd like to bring up that weren't on the agenda?
17:49:59 <afazekas> Periodic master jobs are not in good state
17:50:07 <andreaf_> mtreinish: I'm starting to look in tripleo-ci, I'd like to get tempest running on the overcloud in there
17:50:17 <psedlak> dkranz: what about picking first volunteer(s) based on the amount of errors ... start with those which have just few
17:50:23 <andreaf_> afazekas: sorry go ahead
17:50:30 <mtreinish> afazekas: I just did a buch of changes to what was running in the periodic jobs
17:50:42 <dkranz> psedlak: Sure, except that we can't pick them. They have to volunteer :)
17:50:43 <mtreinish> because the list was really messed up before
17:50:56 <afazekas> We have some peridic jobs (including stress), but according to the log they are failed to start :(
17:50:56 <mtreinish> so I think we may have had some bit rot because they weren't running
17:51:29 <mtreinish> afazekas: yeah I noticed that too, I'll have to dig more into it
17:51:33 <psedlak> dkranz: sure, just we can try to convince them to volunteer a with less errors the resistance could be smaller ;)
17:51:50 <mtreinish> andreaf_: so what is needed to get tempest running there?
17:52:13 <psedlak> dkranz: anyway if any project will agree in the first round we don't need to care about picking one
17:52:14 <andreaf_> mtreinish: well first of all get tempest deployed and then configure it
17:52:18 <afazekas> mtreinish: at the first look the job definition was the same as with the working jobs
17:52:53 <andreaf_> mtreinish: I just started looking into it, I wanted to know if anyone is already on it from this team
17:53:24 * afazekas thinking about  some additional stress jobs  with the know flaky issues
17:53:37 <mtreinish> andreaf_: I haven't been and it hasn't come up on the meeting before. But, I'm sure someone working the tripleo ci stuff is looking at it
17:53:47 <mtreinish> because as I understood it the plan was to run tempest for that
17:54:03 <andreaf_> mtreinish: as tripleo supports multi node environments potentially, there may be some things to be changed to make everything work in multinode scenario as well
17:54:07 <mtreinish> afazekas: can't you just add the decorator the flaky tests?
17:54:34 <dkranz> andreaf_: Yes, part of the work here is to de-devstackize tempest configuration
17:54:43 <dkranz> andreaf_: Which I have also been looking at.
17:55:11 <andreaf_> dkranz: great
17:55:24 <afazekas> mtreinish: I would like to have ~4 jobs with only one or two test on 4-8 threads
17:56:07 <mtreinish> afazekas: it's been a while since I looked at the stress stuff but I think that should be doable with the current framework
17:56:09 <andreaf_> mtreinish: do we have bps on tempest side for this topics (de-devstackize, multinode)
17:56:14 <afazekas> mtreinish: If you have zillion of different things on multiple thread it is difficult to tell what really contributed to the flaky result
17:56:37 <mtreinish> andreaf_: well there was a multinode one for like forever
17:56:42 <afazekas> mtreinish: yes it is
17:57:01 <mtreinish> andreaf_: as for de-devstack stuff I don't think so
17:57:03 <psedlak> andreaf_: https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
17:57:10 <psedlak> #link https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
17:57:20 <mtreinish> but it's something I always look out for in reviews
17:57:48 <afazekas> #link https://bugs.launchpad.net/nova/+bug/1284708
17:58:42 <mtreinish> psedlak: its more about underlying assumptions in tempest about the env
17:58:44 <andreaf_> psedlak: thanks. For tripleo-ci, as we have the config data which is used to setup the environment (like for devstack), I was thinking we could configure tempest based on it - same way we configure services via heat in the overcloud
17:58:56 <dkranz> mtreinish: Right
17:59:29 <dkranz> mtreinish: Like I just discovered tempest expects the admin user to also have admin role in the demo tenant
18:00:07 <dkranz> andreaf_: Yes you could do that.
18:00:08 <mtreinish> ok well we're out of time for today
18:00:15 <mtreinish> thanks everyone
18:00:19 <andreaf_> bye
18:00:20 <mtreinish> #endmeeting