17:00:19 <sdague> #startmeeting qa
17:00:20 <openstack> Meeting started Thu Oct 10 17:00:19 2013 UTC and is due to finish in 60 minutes.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:23 <openstack> The meeting name has been set to 'qa'
17:00:27 <sdague> ok, how's around for the QA meeting?
17:00:38 <ravikumar_hp> hi
17:00:44 <mtreinish> o/
17:00:44 <Anju> hi
17:00:48 <sdague> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_October_10_2013
17:01:24 <sdague> ok, first off, I still need to do the old blueprint purge, a few other things got in the way of that with things like jury duty this week
17:01:38 <sdague> #topic Blueprints (sdague)
17:01:53 <sdague> so I'll work on doing that before next meeting
17:02:04 <sdague> anyone else have blueprint updates to share?
17:02:23 <mkoderer> hi
17:03:06 <dkranz> Here
17:03:55 <giulivo> hi
17:03:58 <sdague> ok, moving on from bluprints
17:04:11 <sdague> dkranz: why don't we jump to your topic
17:04:17 <sdague> #topic Lot's of new negative tests. Do we want that? (dkranz)
17:04:23 <dkranz> sdague: ok
17:04:30 <mkoderer> thats a good one
17:04:46 <dkranz> I am concerned that we have seen a lot of new negative tests.
17:05:01 <dkranz> Last time this happened it was because they are easy low-hanging fruit for new people
17:05:24 <dkranz> This is fine but we should have a clearer line between what it in tempest and what is in unit tests
17:05:39 <giulivo> I added a few "sub-topics" trying to discern which are good and how they should be tagged
17:06:04 <mkoderer> yes for me there are some that are typical unit tests
17:06:14 <dkranz> In an ideal world we would have abstractions where we describe the call to be made and it could be done either as unit or tempest
17:06:32 <dkranz> And there was the idea of "fuzz" testing as well which never went anywhere
17:06:48 <mtreinish> dkranz: I'm fine with negative tests in tempest as long as they're testing api responses.
17:06:51 <sdague> I remember marun actually had this idea at the last summit of tests in the neutron tree that tempest could also run in a real env
17:07:03 <marun> *sigh* yes
17:07:07 <sdague> but I don't think that got any progress
17:07:09 <mtreinish> it goes back to the reason we have api tests in general
17:07:09 <dkranz> sdague: Yes, I have discussed it with others as well
17:07:12 <marun> I did a proof of concept in may but haven't gotten farther
17:07:24 <marun> I should probably post the proof of concept so others can take over
17:07:51 <marun> i'll make sure to have something to show by the summit
17:08:01 <dkranz> I think we would want some kind of yaml or something that describes the actual test data
17:08:30 <mkoderer> marun: could you paste a link to that poc?
17:08:48 <dkranz> The review team could spend all day reviewing negative tests and it might be better to work on making this easier to create and review.
17:08:49 <marun> mkoderer: I never polished it for public consumption, I'm afraid.
17:08:58 <mkoderer> marun: ok I see
17:09:06 <sdague> marun you going to be in HK?
17:09:08 <marun> mkoderer: I'll have to dig it up and get it working again.
17:09:10 <marun> sdague: yes
17:09:22 <sdague> I think this would be a great summit session especially if you had that poc ready
17:09:37 <dkranz> sdague: I think I already added it to proposed on the etherpad
17:09:42 <sdague> dkranz: great
17:09:52 <sdague> honestly, right now, I'm torn on negative testing
17:10:04 <sdague> on the one hand, it's adding coverage
17:10:10 <dkranz> sdague: Well, I think we should have it, but it needs to be easier to write and review
17:10:17 <sdague> but it definitely could be done close to the projects
17:10:39 <dkranz> sdague: The projects could consider tempest to be "closer" than they currently do
17:10:49 <dkranz> sdague: Some are actually making movements to be closer
17:10:50 <sdague> dkranz: so in the short term, I'd propose that we continue to let folks propose it
17:11:03 <sdague> but that we ensure all the negative tests are in seperate files
17:11:12 <sdague> so it would be easier to purge out later
17:11:18 <dkranz> sdague: That seems reasonable
17:11:29 <mkoderer> and they should use the "negative" attr type
17:11:30 <sdague> then we can make final decisions at summit
17:11:33 <sdague> mkoderer: yep
17:11:36 <mtreinish> sdague: sounds like another hacking file update
17:11:40 <dkranz> sdague: But we should also suggest that no one concentrate on that
17:11:40 <sdague> people have been pretty good about it
17:11:51 <sdague> dkranz: sure
17:11:58 <sdague> ok, volunteers for the hacking update?
17:12:02 <dkranz> Once they are up to speed on tempest they should focus on higher-value tests
17:12:18 <afazekas> several negative test are not just simple malformed request, or a  reference to a non existing resource
17:12:31 <giulivo> yeah I still have concerns about that, for instance
17:12:39 <mtreinish> sdague: I guess I'll do it since I brought it up
17:12:41 <giulivo> - some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids
17:12:47 <giulivo> shall we accept both tests?
17:12:51 <sdague> #action mtreinish to update hacking
17:12:58 <giulivo> - some are "traversing" the components, like ensuring we can't attach resources to non existent servers
17:13:03 <dkranz> giulivo: Those cases are malformed requests
17:13:04 <giulivo> shall we accept such a kind of negative?
17:13:05 <sdague> #action mtreinish to update hacking to describe negative tests in dedicated files
17:13:25 <mkoderer> giulivo: yes that a point.. I saw all of them in one review
17:13:31 <giulivo> sorry dkranz why malformed?
17:13:43 <dkranz> giulivo: string too long
17:13:47 <sdague> giulivo: negative is typically more fuzz testing of the API, throwing it garbage
17:14:19 <sdague> afazekas: do you have a good more complicated example?
17:14:22 <giulivo> so if I get it right, non of those is acceptable
17:14:23 <dkranz> I'm fine with those if we don't have to write them long-hand
17:14:30 <dkranz> and review them
17:14:46 <afazekas> Do we want to remove the 'negative' flag from test cases which not just simple fuzz ? Like the auth related ones ?
17:14:48 <sdague> dkranz: right, so this is about some kind of automation on fuzzing
17:14:57 <sdague> afazekas: code example of that?
17:15:00 <mkoderer> https://review.openstack.org/#/c/50217/5
17:15:04 <mkoderer> maybe this one?
17:15:23 <dkranz> sdague: Yes but it is two parts:  declarative test spec, and parameterized randomization
17:15:39 <dkranz> sdague: Not all cases have to be randomly generated
17:15:40 <afazekas> https://review.openstack.org/#/c/50122/ rejected
17:16:03 <dkranz> afazekas: That was rejected for bad spelling, not content
17:16:11 <afazekas> owner_accept
17:16:29 <afazekas> test_image_share_v2_owner_accept
17:17:00 <afazekas> dkranz: the 'rejected' test is not 'negative' sorry
17:17:02 <sdague> dkranz: ok, so that sounds like actually another good topic, and slightly different from marun's one
17:17:28 <dkranz> sdague: Yes. A simple negative test could require setup that is non-trivial
17:17:34 <sdague> yep
17:18:03 <dkranz> So for now perhaps we can just say that we will accept negative tests but not make them a priority
17:18:14 <dkranz> ANd come up with a better plan at the summit
17:18:30 <mkoderer> good plan
17:18:38 <ravikumar_hp> dkranz: yes. and already we have good coverage
17:18:46 <mkoderer> we should create a blueprint after the summit
17:19:08 <mtreinish> mkoderer: did you get your travel approval for hk?
17:19:13 <dkranz> It would be good to have a ml discussion about this prior to the summit to make it more effective
17:19:16 <dkranz> This is a big topic
17:19:20 <afazekas> https://github.com/openstack/tempest/blob/master/tempest/api/compute/test_authorization.py looks like these does not have the negative flag
17:19:31 <sdague> dkranz: yes, agreed
17:19:38 <sdague> dkranz: you want to kick that thread off?
17:19:49 <dkranz> sdague: OK. I will discuss it with marun
17:19:53 <mkoderer> mtreinish: most probably yes
17:19:54 <sdague> great
17:20:25 <sdague> #action dkranz to kick off the thread on openstack-dev list about approaches on negative and shared testing with projects
17:21:06 <dkranz> sdague: Were you thinking tagged with [qa] or not?
17:21:36 <dkranz> sdague: I was going to tag it
17:21:37 <afazekas> https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_access.py
17:21:48 <sdague> yeh, it should be tagged
17:22:16 <dkranz> sdague: I think that's it for that topic
17:23:19 <sdague> ok, great
17:23:37 <sdague> mtreinish: how about you next
17:23:41 <sdague> #topic Elastic Recheck Top issues (mtreinish)
17:23:53 <mtreinish> oh, this is a weekly thing now
17:24:01 <mtreinish> #link http://status.openstack.org/elastic-recheck/
17:24:10 <mtreinish> so we're actually doing really good since er went live
17:24:20 <mtreinish> most of the big bugs have been fixed
17:24:27 <mtreinish> most of the remaining issues are on the neutron side
17:24:48 <mtreinish> there are some new ones popping up that we need to classify still
17:25:13 <sdague> yeh, ER is already a pretty awesome tool
17:25:15 <mtreinish> so if you find a nondeterministic failure and can identify it with a logstash query please submit a review to add it to the query list
17:26:00 <sdague> #link https://github.com/openstack-infra/elastic-recheck
17:26:07 <sdague> for people that haven't seen the code yet
17:26:37 <dkranz> Very cool
17:26:48 <mkoderer> great work
17:26:54 <sdague> ok, lets talk about a related issue, dkranz on whitelisting errors
17:27:06 <sdague> #topic Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
17:27:16 <dkranz> sdague: So the patch is up https://review.openstack.org/#/c/50795/
17:27:46 <dkranz> Many of the non-neutron runs are showing no non-whitelisted errors
17:27:59 <dkranz> I think the next steps are to
17:28:17 <dkranz> 1. Use elastic recheck to track non-deterministic bogus error messages
17:28:26 <dkranz> 2. Communicate all this to the dev list
17:28:35 <dkranz> 3. Start failing builds
17:28:56 <dkranz> 4. Build some kind of momentum to actually get people fixing the bugs
17:28:57 <sdague> that sounds pretty reasonable
17:29:19 <dkranz> I have looked at a lot of errors in the past day and many of them look like real bugs to me
17:29:24 <dkranz> Not just bogus errors
17:29:28 <mtreinish> dkranz: for step 1 you'll need to open bugs for all the non-deterministic bogus error messages you hit
17:29:34 <dkranz> But on one pays attention to the logs when a build succeeds
17:30:01 <dkranz> mtreinish: I was thinking of doing it in a slightly different way
17:30:07 <sdague> dkranz: so something like http://status.openstack.org/elastic-recheck/ for showing the occurance of these might be helpful
17:30:27 <dkranz> Based on what appears in the tempest log
17:30:28 <sdague> just those graphs seemed to make a difference to people in understanding how bad the problems were, and being able to see them go away
17:30:34 <dkranz> sdague: Agreed
17:31:10 <dkranz> So please comment on that patch when you get a chance
17:32:07 <dkranz> sdague: Ideally each group would have some one paying attention to their log errors
17:32:46 <dkranz> The ignoring of these errors is a huge weakness in the tempest methodology
17:33:02 <sdague> yeh, but sun light will help
17:33:05 <dkranz> A test can pass but the system is screwed up afterword
17:33:10 <dkranz> sdague: +1
17:33:24 <sdague> ok, I actually got to run away, dkranz can you take over the rest of the meeting?
17:33:33 <dkranz> sdague: Sure
17:33:37 <dkranz> sdague: Just a sec
17:34:07 <dkranz> We done with this?
17:34:20 <sdague> I think so
17:34:23 <dkranz> #topic Stable Branch Timing
17:34:56 <dkranz> What happened to the topic bot?
17:35:00 <mtreinish> dkranz: I think you need to be sdague to do that
17:35:07 <mtreinish> since he started the meeting
17:35:14 <dkranz> Oh well.
17:35:19 <dkranz> ANyway, next topic
17:35:33 <dkranz> I had proposed branching as late as possible
17:35:44 <dkranz> Any other theories?
17:35:56 <mtreinish> dkranz: I'm fine with that
17:36:16 <dkranz> mtreinish: So when would that be? Release day?
17:36:20 <mtreinish> I don't think we should do it the same time as the other projects
17:36:27 <mtreinish> so next week
17:36:36 <dkranz> mtreinish: RIght, haven't some already done it?
17:36:53 <mtreinish> they've rc'd but I don't think anything is done yet
17:37:19 <dkranz> mtreinish: OK, so the release day is the 17th
17:37:24 <mkoderer> I think branching it as late as possible is the easiest way
17:37:29 <dkranz> We could just do it then
17:37:52 <mtreinish> do you want to do a milestone proposed branch or just go straight to a havana branch?
17:38:12 <giulivo> mtreinish, I was going to propose that, a milestone-proposed branch
17:38:30 <giulivo> but I'd also branch the same day as other projects then, why wait one week more?
17:38:31 <dkranz> mtreinish: Well, if we are waiting until release day I'm not sure why we need milestone-proposed
17:38:48 <mtreinish> yeah that's how I feel too
17:38:59 <giulivo> dkehn, my idea is that people wanting to propose stuff for the stable branch will have to use milestone-proposed
17:39:05 <dkranz> giulivo: Because we are still writing havana tests they will have to be backported once we branch
17:39:05 <giulivo> dkranz, ^^
17:39:06 <mtreinish> ttx: do you have any thoughts on this topic?
17:40:15 <giulivo> dkranz, yeah to solve that very same issue, I'd use a milestone-proposed branch but branch the 17th
17:40:35 <giulivo> (branch our stable, on the 17th)
17:40:55 <mtreinish> giulivo: there's no difference at that point from just doing a stable branch
17:41:01 <dkranz> giulivo: I don't see how that changes anything
17:41:14 <mtreinish> we did a milestone proposed last cycle because we broke something for the rcs when we added a test (I think)
17:41:15 <dkranz> giulivo: The back port rule is our policy
17:41:15 <giulivo> so I think it's important branching the same day as other projects
17:41:28 <giulivo> and the backport rule remains in place for the future additions
17:41:30 <dkranz> giulivo: We can't because they don't all do it on the same day
17:42:25 <giulivo> but tests submitted closer to the branch date, if relevant for the release, can be posted to the milestone and avoid being skipped
17:42:40 <dkranz> I think there is little risk. The 17th is one week from now
17:42:47 <mtreinish> dkranz: lets just say we'll cut the stable/havana right before next week's meeting
17:42:54 <dkranz> mtreinish: Works for me
17:43:43 <dkranz> mtreinish: I can't remember who actually does that. Infra?
17:43:43 <mtreinish> #info stable/havana branch to be cut right before next meeting
17:43:51 <mtreinish> dkranz: sdague will know
17:43:59 <dkranz> OK, any more comments on that?
17:44:54 <dkranz> #topic Design Summit Initial Planning
17:45:17 <dkranz> https://etherpad.openstack.org/icehouse-qa-session-planning
17:45:46 <dkranz> So we have 7 proposals and 13 slots
17:46:04 <dkranz> We don't have to use all of them, but please add to the etherpad if you have ideas.
17:46:34 <ravikumar_hp> dranz: is this the place to add under proposed ?
17:46:55 <dkranz> ravikumar_hp: Yes, the etherpad I pasted above
17:47:06 <ravikumar_hp> http://summit.openstack.org/   ?
17:47:22 <mkoderer> no use the etherpad and put it under "Proposed"
17:47:29 <ravikumar_hp> ok
17:47:39 <dkranz> ravikumar_hp: I think the plan was to do it in the etherpad because it is better for communication
17:48:16 <dkranz> ravikumar_hp: Of course any one who is not participating in these meetings can propose a session on summit.openstack.org
17:48:46 <dkranz> ravikumar_hp: That site is a clumsy tool for collaboration
17:49:09 <dkranz> Any other comments on this topic?
17:49:20 <mkoderer> btw it's my first summit, so if someone want to give me some word how a talk should look like
17:49:24 <ravikumar_hp> dkranz: thanks. i got it
17:49:27 <mkoderer> would be great
17:49:30 <dkranz> mkoderer: No talks!
17:49:48 <mkoderer> I mean I need to prepare something right?
17:49:51 <dkranz> mkoderer: The summit sessions are discussions. YOu can have a few intro slides, but that's it
17:50:08 <mkoderer> dkranz: ok just a few slides good
17:50:13 <dkranz> mkoderer: And try to have enough discussion beforehand so it can be effective
17:50:25 <mkoderer> dkranz: ok cool
17:50:26 <mtreinish> mkoderer: just having an etherpad with an outline of discussion points is normally all that is done up front
17:50:57 <mkoderer> ok fine
17:51:32 <dkranz> OK, any critical reviews before open discussion?
17:51:44 <dkranz> Other than my error log check :)
17:51:56 <mlavalle> dkranz: aren't we going to talk about Neutron? I see it in the agenda
17:52:18 <dkranz> mlavalle: Yes! That had sean's name but please tell
17:52:49 <mlavalle> dkranz: I've been working on this bug https://bugs.launchpad.net/swift/+bug/1224001
17:52:52 <uvirtbot> Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,Fix released]
17:53:02 <dkranz> mlavalle: I saw a number of things in the neutron log files after succesful runs that looked like real bugs
17:53:21 <mlavalle> dkranz: I've been doing this with marun and salv-orlando
17:53:25 <mtreinish> dkranz: for reviews both: https://review.openstack.org/#/c/49559/ and https://review.openstack.org/#/c/49562/
17:53:39 <mtreinish> to split out the unit tests from the jenkins runs
17:54:20 <dkranz> mtreinish: doesn't everything run in jenkins?
17:54:38 <mlavalle> dkranz: salv-orlando proposed a fix to neutron that should help with test_network_basic_ops fails
17:54:49 <mtreinish> dkranz: I should have worded it more clearly. make the unit tests a separate jenkins job only for tempest
17:54:57 <dkranz> mtreinish: Oh, ok
17:55:00 <mtreinish> instead of running it with everything else
17:55:23 <dkranz> mlavalle: That's great
17:55:37 <marun> we definitely need to address errors in the log on otherwise successful runs
17:55:37 <mtreinish> mlavalle: looking at the elastic-recheck graph 1224001 is still happening
17:55:40 <mlavalle> dkranz: I'll test tonight or tomorrow
17:55:52 <marun> it's confusing things when we have failures - hard to see the forest for the trees.
17:56:01 <mtreinish> or do you think it's just an overzealous query
17:56:29 <mlavalle> mtreinish: I don't know. I need to look at it carefully
17:57:04 <mtreinish> mlavalle: http://status.openstack.org/elastic-recheck/
17:57:08 <dkranz> mlavalle: ok
17:57:15 <marun> i've been seeing failures that gerrit attributes to 124001 that are not due to a neutron test failure.
17:57:20 <mlavalle> mtreinish: thanks i'll look at it
17:57:21 <dkranz> We only have a few minutes left
17:57:33 <mlavalle> dkranz: thanks
17:57:42 <mtreinish> marun: it looks for 'tempest.scenario.test_network_basic_ops AssertionError: Timed out waiting for' in the test output
17:57:50 <dkranz> mlavalle, marun : anything else on neutron?
17:57:59 <mlavalle> dkranz: i'm done
17:58:03 <marun> i'm done
17:58:14 <marun> mtreinish: i'll follow up offline about the bug detection
17:58:20 <mtreinish> marun: ok
17:58:22 <dkranz> ok, anything else from any one? We had a full agenda today.
17:58:46 <Anju> dkranz:  can we renamed the Duplicate exception to Conflict ?
17:58:59 <Anju> dkranz:  can we rename the Duplicate exception to Conflict ?
17:59:28 <mkoderer> let's discuss it after the meeting?
17:59:29 <Anju> as 409 depicts Conflict
17:59:40 <dkranz> Yes, we are out of time.
17:59:45 <Anju> sure
17:59:53 <dkranz> #endmeeting
18:00:13 <mtreinish> fungi, clarkb, jeblair: ^^^
18:00:13 <dkranz> mtreinish: Can sdague end the meeting now or is he really gone?
18:00:22 <mtreinish> sdague is away how do we end the meeting
18:00:28 <clarkb> there is a timeout in meetbot
18:00:38 <clarkb> it is 60 minutes, so anyone can end it after that timeout
18:00:50 <clarkb> try it again
18:00:57 <dkranz> #endmeeting