17:00:08 <oomichi> #startmeeting qa
17:00:09 <openstack> Meeting started Thu Aug 18 17:00:08 2016 UTC and is due to finish in 60 minutes.  The chair is oomichi. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:13 <openstack> The meeting name has been set to 'qa'
17:00:31 <oomichi> who are here today?
17:00:38 <jlvillal> o/
17:00:47 <mtreinish> o/
17:02:12 <oomichi> jlvillal: mtreinish: hi :)  lets wait for more people a little bit
17:02:27 <auggy> o/
17:02:47 <oomichi> auggy: hi, ok let's start a meeting
17:02:53 <oomichi> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_August_18th_2016_.281700_UTC.29
17:03:01 <oomichi> ^^^ is today agenda
17:03:06 <andreaf> o/
17:03:10 <jordanP> not much:)
17:03:17 <oomichi> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:03:35 <oomichi> andreaf: jordanP: hi, good for us :)
17:03:49 <oomichi> there are few open qa-specs
17:04:14 <oomichi> the first one is  https://review.openstack.org/#/c/173334/
17:04:38 <oomichi> andreaf reviewed already just before this meeting, thanks
17:04:44 <oomichi> I will check the comment later
17:04:52 <andreaf> yes I just +2 it as well
17:05:20 <oomichi> andreaf: ah, that is owner, OK, will change it;)
17:05:44 <oomichi> andreaf: I will start creating a prototype later
17:06:05 <oomichi> the second is https://review.openstack.org/#/c/349730/
17:06:16 <andreaf> oomichi: there were already some patches by jswarren and dwallek you may want to start from those
17:06:44 <oomichi> andreaf: oh, I didn't know that, can you provide the links of patches?
17:07:46 <andreaf> oomichi: #link https://review.openstack.org/#/q/status:open+branch:master+topic:bp/tempest-resources
17:08:18 <oomichi> andreaf: thanks
17:08:40 <andreaf> Regarding the spec #link https://review.openstack.org/#/c/349730/
17:09:05 <oomichi> andreaf: yeah, please go ahead
17:09:21 <luzC> o/
17:09:22 <andreaf> The idea there is to define resources that we create at class level as fixtures
17:10:06 <andreaf> since there is no way to use fixtures in class level fixtures atm I also prototyped https://review.openstack.org/#/c/351871/
17:10:59 <oomichi> andreaf: the class level means test class ?
17:10:59 <andreaf> I got negative feedback on the idea from lifeless, basically he says class level things are fragile and don't work well for parallelism
17:11:06 <andreaf> oomichi: yes
17:11:20 <andreaf> setUpClass and tearDownClass are class level fixtures
17:11:50 <oomichi> andreaf: Test developers need to know what resources are created on each used test class
17:11:56 <andreaf> they are classmethods and they work on class, so they can provide a shared context for all test methods defined by a test clas
17:12:49 <andreaf> oomichi: yes that's part of the cleanup we want to do - to make it obvious what resources are created
17:13:31 <oomichi> andreaf: and all these resources are removed on the cleanup, that seems good merit
17:14:18 <oomichi> andreaf: my concern is tests will create unnecessary resources also just for using small amount of resources
17:14:24 <jordanP> I think the whole point is to share some utility functions (that setup the environment: create projects, users etc...) between classes or plugins without using inheritence
17:14:25 <andreaf> oomichi: so I'm a bit struggling on this since it's an investment in a direction that lifeless recommends not to follow
17:15:06 <oomichi> jordanP: yeah, I tend to prefer that without inheritence at this time
17:15:09 <andreaf> oomichi: on the other side though using testresources would me stop relying on class level fixtures and attributes for good, which would be a major rework
17:15:34 <mtreinish> andreaf: it's a lot of work...
17:15:43 <mtreinish> multiple cycles at least
17:15:55 <mtreinish> at least when I looked at it before
17:16:13 <jordanP> yeah, a lot of work, and I am not sure it's brings a big benefit
17:17:16 <oomichi> andreaf: to be honest, I could not get the idea we are discussing now from the spec
17:17:21 <andreaf> jordanP: the main benefit would be to stop caged on the class context, and being able to select tests in a different way - the test scheduler could optimise better, based on resource utilisation
17:17:36 <andreaf> jordanP: however none of those seem to me like major benefits for tempest
17:18:04 <oomichi> andreaf: the "Proposed change" seems stopping writting in the middle
17:18:21 <oomichi> andreaf: like line 100 or later
17:19:11 <andreaf> oomichi: yes I need to update the spec - but I started prototyping class level fixtures and I wanted some feedback on that so I can include in the spec the direction we are taking, and why, with alternatives etc
17:19:58 <mtreinish> andreaf: yeah, that was my thinking too. It'd also a fundamental change in how we expect tempest to be run. There are lots of baked in assumptions that the minimum parallel unit is a class
17:20:03 <oomichi> andreaf: yeah, that is good way to move forward. I can understand easily from the code
17:20:40 <auggy> is it worth looking at how other OpenStack projects have solved this problem (if they have)?
17:20:49 <mtreinish> it's more than just resources
17:21:15 <mtreinish> auggy: it's a bit of a special problem for tempest given it working with real clouds and using the unit test framework
17:21:54 <mtreinish> auggy: other projects either use test resources (keystone client and oslo.db for example) or just fixtures
17:22:01 <mtreinish> depending on what they're trying to do
17:22:19 <auggy> ok, then i probably don't fully understand the problem :) i understood it as tempest needs to set up different things based on different configurations at the test class level and currently that's difficult to do
17:22:33 <mtreinish> but in all those cases they were also building frameworks from scratch which is a different problem
17:22:45 <andreaf> mtreinish: one fear is that eventually class level fixtures will be deprecated - however that will take a long time if it happens - and we could still have our own TestSuite with support for them
17:23:37 <mtreinish> andreaf: heh, I find that extremely unlikely. But then you never can tell with backwards compat and python :P
17:24:27 <mtreinish> auggy: it's actually even simpler than that we're trying to make some of the magic we're using through classmethods, atrributes, and inheritence more explicit in the actual test classes where things are run
17:24:38 <mtreinish> auggy: and trying to figure out the best mechanism for doing that
17:24:41 <auggy> ah ok
17:24:44 <jordanP> yep
17:25:05 <auggy> ah ok, that makes sense, thanks
17:25:24 <auggy> i always disliked Python's heavy use of inheritance, are there patterns for Mixins? would that be an option?
17:26:00 <oomichi> auggy: yeah, I also don't line deep inheritance, that is unreadable
17:26:17 <andreaf> auggy: we are trying to use fixtures to avoid class inheritance
17:26:49 <andreaf> auggy: since fixtures are easy to use, and isolated and self-cleaning, test don't need to rely on that functionality being an a super class
17:26:57 <auggy> ah right ok
17:27:12 <auggy> i've only used the fixtures through nova and it's all pretty heavy inheritance
17:27:23 <mtreinish> auggy: if you want a good example of what we're trying to fix see: https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L411-L462
17:27:29 <auggy> thanks for clarifying, hope i didn't derail the conversation too much :)
17:27:30 <mtreinish> and try to figure out exactly what that test is doing
17:27:50 <auggy> yeah i've been working with Nova's functional tests hahahaa
17:27:58 <andreaf> good luck with that :D
17:28:03 <oomichi> mtreinish: hehe, nice sample
17:28:14 <lifeless> andreaf: did you get time to play with testresources ?
17:28:33 <mtreinish> oomichi: well, I've been trying to unroll that in my spare time for the past week (and not made it very far)
17:29:41 <andreaf> lifeless: only a bit - I think it could work - but it would be a major change in how we do things in Tempest
17:29:51 <oomichi> mtreinish: I see, thanks:)
17:30:00 <andreaf> lifeless: and I'm not sure the benefits for Tempest are enough to justify the effort
17:30:45 <lifeless> andreaf: I'm quite a bit confused by the expectation that it would be a tonne of work; it seems very mechanical to me
17:30:52 <andreaf> lifeless: we tend to have less heavier more static tests compared to a unit test suite - and I believe for that scenario relying on the class unit is not so bad
17:31:19 <lifeless> andreaf: perhaps mid-meeting isn't the right time to analyse it; but - I'd like to
17:31:29 <lifeless> I do -not- want to be a blocker on improvements
17:32:10 <oomichi> andreaf: lifeless: can we discuss it deeply later ?
17:32:34 <oomichi> the time went in the first topic ;)
17:32:50 <andreaf> oomichi, lifeless: sure - but we're towards the end of my dey, so maybe tomorrow
17:33:10 <lifeless> oomichi: course
17:33:11 <oomichi> andreaf: thanks
17:33:16 <oomichi> ok, let's move on
17:33:24 <oomichi> #topic Tempest
17:33:33 <oomichi> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
17:34:11 <oomichi> there are still many active patches
17:34:22 <jordanP> https://review.openstack.org/#/c/351426/
17:34:25 <oomichi> are there patches we need to discuss now
17:34:45 <jordanP> I don't think we should remove any Heat test
17:34:48 <oomichi> jordanP: oh, go ahead
17:35:09 <jordanP> but gmann disagrees
17:35:26 <andreaf> jordanP: heat is not in scope for Tempest - at least not right now
17:35:33 <jordanP> why not ?
17:35:42 <mtreinish> jordanP: it's in the set of things out of scope in tempest: http://docs.openstack.org/developer/tempest/test-removal.html#tempest-scope
17:35:49 <oomichi> jordanP: Tempest scope is for six cores
17:36:01 <jordanP> but I like heat :)
17:36:17 <oomichi> jordanP: and if defcore extends cores, tempes needs to contain a new core test
17:36:36 <oomichi> jordanP: heh, that is different
17:36:42 <luzC> @jordanP +1
17:36:43 <jordanP> "be removed assuming there is equivalent testing elsewhere"
17:36:59 <jordanP> I am not sure there is
17:37:15 <mtreinish> jordanP: heat has had there own functional test suite for a while
17:37:21 <mtreinish> s/there/their
17:37:51 <jordanP> yeah but I remember a bug in Heat that only Tempest discoved
17:37:55 <oomichi> jordanP: mtreinish: yeah, as https://review.openstack.org/#/c/335740/ pointed
17:37:57 <jordanP> a couple of months ago
17:38:32 <jordanP> (https://bugs.launchpad.net/heat/+bug/1546201)
17:38:32 <openstack> Launchpad bug 1546201 in heat "Race condition: boot a server on a network created by Heat" [High,Fix released] - Assigned to Crag Wolfe (cwolfe)
17:38:34 <oomichi> jordanP: after that, the migration happened according to the patch
17:38:44 <jordanP> how are we sure that we are not losing coverage ?
17:38:54 <andreaf> jordanP: heat tests are not run in the integrated gate anyways
17:39:06 <andreaf> jordanP: I think they are in the layer4 job only
17:39:27 <mtreinish> jordanP: we rely on heat for that judgement
17:39:29 <jordanP> yes they run in the layer4 job
17:39:35 <oomichi> jordanP: the coverage of non-cores should be own on each project, not qa-team
17:39:57 <mtreinish> jordanP: the heat coverage in tempest is very minimal fwiw
17:40:23 <jordanP> ok, got it
17:40:41 <jordanP> we can move forward
17:40:57 <oomichi> are there more items related to tempest?
17:41:19 <oomichi> ok, let's move on
17:41:31 <oomichi> #topic devstack + grenade
17:41:44 <oomichi> #link https://review.openstack.org/#/q/project:openstack-dev/devstack+status:open
17:41:53 <oomichi> #link https://review.openstack.org/#/q/project:openstack-dev/grenade+status:open
17:42:20 <oomichi> there are small amount of patches on grenade
17:42:28 <mtreinish> oomichi: there always are
17:42:39 <mtreinish> it's a very low activity project
17:42:41 <oomichi> mtreinish: yeah
17:43:07 <oomichi> isn't enough to review the patches now?
17:43:48 <oomichi> the first one seems easy, but that takes time
17:43:57 <oomichi> #link https://review.openstack.org/#/q/project:openstack-dev/grenade+status:open
17:44:01 <oomichi> ops wrong
17:44:17 <oomichi> #link https://review.openstack.org/#/c/296003/
17:45:23 <oomichi> just small comment, are there any items related to devstack and grenade?
17:45:42 <mtreinish> oomichi: well since the last meeting neutron is now the devstack default
17:45:56 <mtreinish> that's kinda big
17:46:04 <andreaf> \o/
17:46:05 <jordanP> omg, nova-net is deprecated ? ^^
17:46:06 <oomichi> mtreinish: good news:)
17:46:32 <mtreinish> jordanP: yeah nova re-deprecated it a while ago
17:46:48 <jordanP> (yeah, just trolling :p)
17:46:49 <oomichi> that takes much time
17:46:52 <mtreinish> haha
17:47:17 <oomichi> do we have nova-network specific job?
17:47:25 <mtreinish> oomichi: the jobs have not changed
17:47:53 <mtreinish> oomichi: but if you run devstack yourself you have to manually enable n-net and disable neutron
17:48:10 <oomichi> mtreinish: h, I see. I misunderstood all jobs are changed
17:49:02 <oomichi> ok, that is good for getting feedback from uesrs
17:49:35 <oomichi> if not having any items, let's move on
17:49:44 <oomichi> #topic openstack-health
17:50:10 <oomichi> #link https://review.openstack.org/#/c/296003/
17:50:30 <mtreinish> oomichi: the same grenade patch?
17:50:35 <oomichi> mtreinish: is the node_provide meaning public cloud?
17:50:35 <jordanP> :p
17:50:43 <andreaf> oomichi: I have a couple of patches for subunit2sql, which is related - but it's actually an openstack-infra project, so not sure I should mention them here?
17:50:46 <oomichi> #link https://review.openstack.org/#/c/356060/
17:50:54 <oomichi> oops, sorry about that
17:51:03 <andreaf> oomichi: https://review.openstack.org/#/q/status:open+project:openstack-infra/subunit2sql+branch:master+topic:story/2000698
17:51:14 <mtreinish> oomichi: node_provider is the cloud region the job is running
17:51:15 <andreaf> oomichi: anyways - giving it a shot ;)
17:51:22 <mtreinish> oomichi: so rax iad, osic, etc
17:51:49 <mtreinish> andreaf: yeah it's fine to talk about here. It's always straddled the fence
17:51:54 <oomichi> mtreinish: that is good information, these days, gate was not in good condition
17:52:12 <oomichi> mtreinish: these info will be helpful for that condition
17:52:22 <oomichi> andreaf: please go ahead
17:52:31 <andreaf> mtreinish: ok - well I just wanted to get some reviews on them #link https://review.openstack.org/#/q/status:open+project:openstack-infra/subunit2sql+branch:master+topic:story/2000698
17:52:53 <mtreinish> andreaf: yeah I've been meaning to take another look at those
17:53:11 <andreaf> I added unit test coverage, fixed a bug and added a new config option to drop the test_attr_prefix before adding metadata into the test_metadata table
17:53:15 <mtreinish> oomichi: right, that's why we landed a bunch of improvements to the test page recently: http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image?end=2016-08-16T15:00:00.000Z
17:53:24 <mtreinish> oomichi: make it easier to find common trends in failures
17:53:31 <oomichi> mtreinish: ++
17:53:32 <mtreinish> it's still not perfect yet but it's getting better
17:53:45 <oomichi> mtreinish: yeah, agree. nice direction
17:53:45 <mtreinish> masayukig is working on fixing some of the graphing we have there now
17:53:56 <andreaf> mtreinish: oh, nice
17:54:10 <mtreinish> and I think visualizing common pieces of metadata somehow will be a good future step after that
17:54:12 * andreaf goes and merges those patches in his own o-h
17:55:16 <andreaf> mtreinish: I think it would be really great to get distributions of metadata keys over values, like elastic search does
17:55:25 <mtreinish> andreaf: there are some performance things I need to fix, I have a better strategy for caching in mind that should make it more responsive
17:55:29 <mtreinish> andreaf: yeah that's what I was thinking
17:55:42 <mtreinish> I'm just not sure how to do that in js :)
17:55:51 <andreaf> :D
17:56:00 <oomichi> mtreinish: heh
17:56:11 <oomichi> ok, let's move on if not having more
17:56:23 <oomichi> #topic Critical reviews
17:56:34 <oomichi> are there any patches we need to concentrate on?
17:57:24 <mtreinish> #link https://review.openstack.org/#/q/Id65ebae73b28da7185cb349b714b659af51ef77f,n,z
17:57:41 <mtreinish> which switches us to use qcow by default in devstack
17:57:57 <mtreinish> there is a corresponding tempest change, but there are grenade failures I need to figure out
17:58:07 <jordanP> not a patch but we've been lagging a bit in term of reviewing lately, imo
17:58:30 <jordanP> is that your feeling too ?
17:58:45 <jordanP> mtreinish, yes that's a good patch, I never liked the ami format
17:58:54 <oomichi> mtreinish: yeah, that is a good one
17:59:12 <oomichi> I will +A after passing the gate
17:59:45 <oomichi> jordanP: yeah that is a little active now
18:00:03 <oomichi> #link http://stackalytics.com/report/contribution/tempest/30
18:00:17 <mtreinish> jordanP: yeah I get that sense too. I think it's because we have a bunch of inactive cores
18:00:26 <oomichi> ops, time is coming
18:00:33 <oomichi> #endmeeting