22:01:05 <mtreinish> #startmeeting qa
22:01:06 <pennerc> o/
22:01:06 <openstack> Meeting started Thu Mar  5 22:01:05 2015 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:11 <openstack> The meeting name has been set to 'qa'
22:01:16 <mtreinish> Hi, who's here today?
22:01:32 <riwinters> hi
22:01:39 <dkranz> hi
22:01:43 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_March_5th_2015_.282200_UTC.29
22:01:43 <masayukig> o/
22:01:48 <mtreinish> ^^^ Today's agenda
22:02:08 <gmann> hi
22:02:45 <dtroyer> o/
22:03:00 <mtreinish> sdague, jogo: courtesy ping
22:03:04 <mtreinish> ok let's get started
22:03:14 <andreaf> o/
22:03:18 <mtreinish> #topic Tempest test removal proposal reviews
22:03:40 <pennerc> o/
22:03:46 <mtreinish> so this meeting is the first meeting with tempest test removals proposed per:
22:03:49 <mtreinish> #link https://wiki.openstack.org/wiki/QA/Tempest-test-removal
22:04:00 <mtreinish> there are only 2 and I expect them to be quick
22:04:02 <jogo> o/
22:04:17 <mtreinish> the 2 removals proposed are:
22:04:20 <mtreinish> #link https://etherpad.openstack.org/p/tempest-test-removals
22:04:50 <mtreinish> the first one on the list is from stevebaker:
22:05:01 <mtreinish> tempest.scenario.orchestration
22:05:09 <stevebaker> \o
22:05:20 <mtreinish> which is removing the heat scenario tests, which have been skipped since the begining of the cycle
22:05:35 <stevebaker> seems uncontroversial to remove
22:05:37 <mtreinish> (so step 2 doesn't really apply)
22:05:42 <mtreinish> stevebaker: yep
22:05:54 <mtreinish> does anyone have an objection to removing this?
22:06:22 <sdague> nope, lets do it!
22:06:22 <masayukig> mtreinish: nope
22:06:26 <dkranz> no
22:06:27 <jogo> why have they been skipped?
22:06:32 <jogo> (and no objection)
22:06:57 <stevebaker> jogo: transient failures at the time due to slow nodes. we're not seeing the same issue in heat functional tests
22:07:00 <mtreinish> ok I'll remove my -2 and we can merge it
22:07:18 <mtreinish> jogo: it says bug 1374175
22:07:19 <openstack> bug 1374175 in heat "test_server_cfn_init failed in gate-tempest-dsvm-neutron-heat-slow: AssertionError: Timed out waiting for 10.1.0.4 to become reachable" [Undecided,Incomplete] https://launchpad.net/bugs/1374175 - Assigned to Steven Hardy (shardy)
22:07:21 <stevebaker> can we quickly talk abou removal of heat cli test?
22:07:32 <jogo> stevebaker: ahh makes sense
22:07:46 <mtreinish> stevebaker: can we wait until after the other removal on the etherpad?
22:08:49 <stevebaker> sure
22:09:07 <mtreinish> the other proposed removal from the etherpad is from sdague:
22:09:23 <mtreinish> tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor and tempest.api.compute.admin.test_flavors_negative.FlavorCreateNegativeTestJSON
22:09:34 <sdague> yeh, it was updated this morning to delete the whole test file
22:09:57 <sdague> as I now have the ability to test the user/admin split in the nova patch series as well
22:10:08 <mtreinish> oh, ok I didn't see that but that doesn't make much of a difference
22:10:18 <sdague> the Depends-On link was updated to the correct point in the nova patch series
22:10:27 <sdague> yeh, should be the same
22:10:30 <mtreinish> sdague: that link doesn't look correct
22:10:34 <mtreinish> it doesn't go anywhere
22:10:51 <sdague> apparently, truncated
22:10:53 <sdague> let me fix
22:11:04 <mtreinish> sure it's just missing the "I" in front
22:11:22 <sdague> yeh
22:11:24 <sdague> it was
22:11:32 <sdague> ok, updated
22:11:45 <mtreinish> but aside from the typo based on the numbers these tests seem like something else that's not really controversial
22:12:11 <mtreinish> especially given none of these tests have failed once since Nov. in the gate
22:12:25 <mtreinish> (according to the numbers in the etherpad)
22:12:38 <mtreinish> does anyone have an objection to removing the flavor negative tests?
22:13:05 <andreaf> no, +1 from me
22:13:06 <masayukig> nope
22:13:09 <dkranz> no
22:13:19 <gmann> no. +1
22:13:41 <mtreinish> ok, then I'll remove my -2 and let it move forward
22:14:01 <mtreinish> stevebaker: ok, what did you want to ask about the heat cli tests?
22:14:44 <sdague> also, the heat test removal needs a rebase, because of idempotent_id land
22:14:46 <stevebaker> rather than creating a new python-heatclient functional job I'd like to move the test to the heat functional job, and make python-heatclient gate on that
22:15:11 <stevebaker> heat already depends on python-heatclient
22:15:52 <stevebaker> that was all, I just wanted to mention in case there are objections
22:16:03 <sdague> that seems confusing to me, but I think how you build those jobs is pretty much the heat team's call
22:16:15 <mtreinish> stevebaker: well how you gate is up to you guys. BUt that seems weird to me, unless you are running the heats tests on every heatclient patch
22:16:34 <sdague> I think the important thing is to get the cli calls out of tempest, because they don't really fit there
22:16:35 <mtreinish> but it's your call
22:16:36 <stevebaker> we would, they would co-gate
22:16:47 <mtreinish> stevebaker: ok
22:17:31 <mtreinish> ok then does anyone else have something to discuss on tempest test removals? Otherwise let's move on
22:18:25 <mtreinish> #topic Devstack
22:18:48 <mtreinish> dtroyer: so anything new and exciting from devstack this week?
22:19:08 <dtroyer> I don't have much on my list as I've been focused on catching up with other things this week
22:19:20 <stevebaker> I've just proposed to remove lib/dib https://review.openstack.org/#/c/161919/
22:19:54 <mtreinish> stevebaker: hmm I think I remember reading a spec from adam_g doing something similar at some point
22:20:00 <mtreinish> I'll take a look after the meeting
22:20:07 <mtreinish> dtroyer: ok, sure
22:20:17 <stevebaker> it was a dead-end experiment which was never used
22:20:25 <adam_g> i actually had a spec to formalize our use of it, but at this point i think its okay to remove
22:20:35 <mtreinish> adam_g: heh, ok
22:20:43 <adam_g> ironic is actually moving away from DIB built ramdisks so...
22:20:47 <sdague> adam_g: it probably should be an external plugin if you want to use dib there
22:20:50 <stevebaker> pip install diskimage-builder seems to do the job
22:20:58 <adam_g> sdague, yea
22:21:09 <sdague> stevebaker: looks great, thanks!
22:21:23 <sdague> yay for deleting stuff from devstack :)
22:21:36 <stevebaker> indeed!
22:21:38 <adam_g> speaking of devstack plugins, ive been playing with moving the ironic bits out of devstack and into a plugin.. do we have a documented process for deprecating and removing the devstack code?
22:22:10 <mtreinish> adam_g: not really, it's been more of just do it :)
22:22:41 <adam_g> okay. i guess we can keep it around with deprecation warnings everytime its sourced for a while
22:23:11 <sdague> I think that doing it for something that has grenade jobs on it is going to be hard until grenade is modularized for it
22:23:29 <mtreinish> sdague: yeah we've hit that already on a few things
22:23:47 <sdague> but, my brain power has all been on nova this week, so I haven't thought that through yet
22:24:05 <dtroyer> adam_g:  we do have DEPRECATED_TEXT that is displayed at the end of stack.sh, if you want to add something to that it should be easily seen.
22:24:23 <adam_g> dtroyer, nice, thanks
22:24:54 <mtreinish> the only thing I have on devstack this week is jogo and I have been on the quest to get our gate runtime under control.
22:25:16 <adam_g> also, for anyone interested in devstack plugins: https://review.openstack.org/#/c/161535/ deals with installation of distro packages required by plugins. didn't see anything in the spec about that
22:25:41 <mtreinish> mostly around pre-caching wheels in nodepool, because the wheel compilation takes a non-negligible amount of time
22:26:00 <sdague> mtreinish: ++
22:26:31 <jogo> non-negligible ~= 5 minutes
22:26:34 <mtreinish> we're consistently seeing devstack take ~20+ min at this point, which is no good
22:26:44 <sdague> yeh, numpy is 3 minutes iirc
22:27:19 <mtreinish> ok, does anyone have anything else to discuss on devstack?
22:27:31 <sdague> nope, I need to drop as well, see folks tomorrow.
22:27:47 <mtreinish> o/
22:28:23 <mtreinish> #topic Grenade
22:28:44 <mtreinish> I'm not expecting anything for this, things have been pretty quiet on the grenade front
22:28:52 <mtreinish> jogo, dtroyer: anything to add? :)
22:29:33 <dtroyer> nope
22:29:45 <mtreinish> ok does anyone else have anything on grenade?
22:30:40 <mtreinish> #topic Tempest Configuration Doc (mtreinish)
22:30:51 <mtreinish> ok this is one of the leftover from last week
22:31:10 <mtreinish> I wanted to bring up that we've started a tempest configuration guide in the docs
22:31:18 <mtreinish> #link http://docs.openstack.org/developer/tempest/configuration.html
22:31:36 <mtreinish> right now it's pretty sparse and just contains an overview on how to do the auth/creds setup
22:32:07 <mtreinish> but I think considering the number of questions I've been fielding from people struggling with this the past few weeks it would be good if we could expand this
22:32:34 <mtreinish> I was hoping people would volunteer to help tackle a bit and add on to it
22:33:06 <dkranz> mtreinish: do you have a list of options causing trouble based on what you were asked?
22:34:04 <mtreinish> dkranz: most of the confusion I've seen was around enabled features and available services (which seemed strange to me because those should be the easy ones)
22:34:26 <mtreinish> oh and the ec2 configuration throws everyone for a loop
22:34:53 <dkranz> mtreinish: heh
22:34:56 <andreaf> mtreinish, dkranz: as part of the ssh auth bp we have some work on configuration, so we'll update that bit (timeouts, ssh credentials and so)
22:35:54 <mtreinish> andreaf: cool
22:36:17 <mtreinish> ok does anyone else have something on this topic?
22:36:31 <gmann> andreaf: +1, ppl get confused for ssh options we have currently. i saw bug for that.
22:36:48 <masayukig> mtreinish: how about bringing it to ML(openstack or openstack-dev)? I don't know how many people are using Tempest, though.
22:37:02 <mtreinish> masayukig: +1 that's a good idea
22:37:24 <mtreinish> #action mtreinish to bring up the new configuration guide on the ML
22:37:59 <dkranz> masayukig: I think there are many, many people using tempest
22:38:00 <mtreinish> andreaf: would you be opposed to documenting what you have to do now for ssh?
22:38:16 <masayukig> dkranz: great! :)
22:39:13 <andreaf> mtreinish: I can give it a go
22:39:20 <mtreinish> andreaf: cool, thanks
22:39:39 <mtreinish> #action andreaf to work on adding ssh section to the config doc
22:39:48 <mtreinish> ok let's move on
22:39:57 <mtreinish> #topic new tag for interop tests (hogepodge)
22:40:02 <mtreinish> hogepodge: around?
22:40:11 <hogepodge> o/
22:40:33 <mtreinish> hogepodge: the floor is yours
22:40:41 <hogepodge> I'd like to propose a new tag for tests, working on a blueprint for it right now.
22:41:23 <hogepodge> I feel that there's a need for strict api testing that the current set of api tests doesn't meet.
22:42:03 <hogepodge> It should be possible to just point tempest at a cloud with only an endpoint and credentials, and maybe a minimum of pre-existing resources (network id and image id, for example) and check for api functionality.
22:42:35 <dkranz> hogepodge: Isn't that what tempest does now? Coverage is a different issue.
22:42:36 <hogepodge> My proposal is a new test tag, [interop] which has the requirements of testing api endpoints with only minimum configuration required.
22:43:02 <hogepodge> dkranz lots of api tests want to have admin access for a number of different reasons
22:43:11 <dkranz> hogepodge: As I said earlier in IRC, that is going to cover most non-admin tests
22:43:23 <dkranz> hogepodge: yes, but we are fixing that
22:43:39 <andreaf> hogepodge, dkranz: sounds like "smoke" revisited
22:43:40 <mtreinish> hogepodge: for the most part if you don't provide admin it should just skip those. (there are bugs there though, we're working through them)
22:44:06 <dkranz> andreaf: I think it is a little different. smoke is supposed to be a "minimal" set of testsw
22:44:19 <dkranz> andreaf: and we need to revisit that as well
22:44:27 <hogepodge> if there's overlap, or other spaces that's handled by this, then that's great.
22:44:45 <mtreinish> dkranz: well that's one of the goals for the code sprint to clean up the usage of the smoke tag.
22:44:48 <hogepodge> The defcore committee is seeing otherwise, though.
22:44:51 <dkranz> hogepodge: no, I think a new tag is needed but it should be "not interop"
22:44:51 <mtreinish> But I agree this might be a superset on smoke
22:45:01 <mtreinish> err, actually probably just an overlapping set
22:45:11 <mtreinish> because smoke will include scenario and this probably won't
22:45:29 <dkranz> hogepodge: there are a few tests that are non-admin but depend on some configuration that is hard for a user to know.
22:45:36 <dkranz> hogepodge: that is how I would interpret this
22:46:07 <hogepodge> dkranz: that's part of it. An endpoint, identity, and modest resources should really be sufficient.
22:46:17 <dkranz> hogepodge: the interop tests are those that do not depend on values in tempest.conf that can be different for different deployments
22:46:22 <hogepodge> I don't want to chase an idea that doesn't have community support, though.
22:46:37 <dkranz> hogepodge: I don't think it necessarily doesn't have support
22:47:02 <mtreinish> hogepodge: well I think write up a spec to flush out the idea. Then we can work through it when there is a solid idea out there
22:47:10 <dkranz> hogepodge: IMO, only a small number of tests need to be tagged, and then if you don't provide admin credentials, only what you want will run
22:47:29 <dkranz> mtreinish: +1
22:48:01 <gmann> hogepodge: dkranz : mtreinish  i am confused about interops requirement, is it tests with minimum conf or a set of functionality to test irrespective of how much conf needed
22:48:12 <hogepodge> dkranz: One thing I'd like to see is a pool of tests that can be used to really define interoperability. The defcore and committee and board need guidance on what's important to test and how to measure compatibility between deployments.
22:48:40 <mtreinish> hogepodge: isn't that what defcore is supposed to define?
22:49:09 <hogepodge> mtreinish: Right now the committee looks at api tests, ranks them against a criteria, then determines the list.
22:49:14 <dkranz> mtreinish: yes but I think the presumption is it can be extracted from existing tempest tests
22:49:15 <mtreinish> gmann: I think that's why we need a spec to clearly outline the requirements here
22:49:51 <hogepodge> mtreinish: we'd like more involvement from the PTLs and community members on what's important. The assumption is that if an api test exists it expresses importance.
22:50:09 <mtreinish> the tests should just be the verification mechanism. I don't think they should be used to define it
22:50:30 <mtreinish> but anyway I don't want to get too far off track
22:50:37 <mtreinish> does anyone have anything else on this?
22:50:39 <hogepodge> mtreinish: it's a chicken and egg problem.
22:50:52 <dkranz> mtreinish: I think we should assume all non-admin tests cover things that are supposed to be interoperable and identity exceptions
22:51:16 <dkranz> mtreinish: I really don't understand why there is controversy here, if there is any
22:51:35 <mtreinish> hogepodge: not really, because we can add a test at any point.
22:51:39 <mtreinish> but let's move on
22:51:45 <andreaf> dkranz: there are tests which use admin credentials for preparation, but they do not test an admin api, and they might belong to interop
22:51:52 <mtreinish> because we're at <10mins
22:52:29 <mtreinish> #topic Bugs
22:52:45 <mtreinish> #link https://etherpad.openstack.org/p/Tempest-bug-report
22:53:05 <mtreinish> gmann: anything new on bugs this week?
22:53:26 <mtreinish> masayukig: you had the rotation this week right?
22:53:30 <gmann> mtreinish: nothing much. masayukig had triage this week
22:54:23 <masayukig> mtreinish, gmann: yes. I've done.
22:54:58 <mtreinish> ok anything else on bugs? Otherwise let's move one?
22:55:02 <mtreinish> s/one/on
22:55:13 <gmann> and people volunteered for triage rotation. we have triage rotation fill till Apr 6
22:55:22 <gmann> that's all from my side
22:55:31 <mtreinish> yeah I noticed that, I was surprised :)
22:55:39 <gmann> :)
22:55:48 <mtreinish> #topic Critical Reviews
22:56:06 <mtreinish> ok does anyone have a review they'd like to get additional eyes on?
22:56:18 <andreaf> https://review.openstack.org/#/q/topic:+bp/multi-keystone-api-version-tests+status:+open,n,z
22:57:12 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/test-accounts-continued,n,z
22:57:30 <andreaf> mtreinish: you were faster :)
22:57:46 <andreaf> #link https://review.openstack.org/#/q/topic:+bp/resource-cleanup+status:open,n,z
22:57:54 <andreaf> #link https://review.openstack.org/#/q/topic:+bp/multi-keystone-api-version-tests+status:+open,n,z
22:58:30 <andreaf> I actually had a question on the multi-auth bp, but perhaps after the meeting as we're short on time
22:58:48 <gmann> andreaf: I had look yesterday on keystone one. ll review today
22:58:57 <mtreinish> andreaf: sure we can pick that up in -qa after the meeting
22:59:00 <andreaf> gmann: thanks
22:59:06 <mtreinish> ok are there any other reviews to bring up?
22:59:26 <mtreinish> #link https://review.openstack.org/#/c/158017/
22:59:45 <mtreinish> that's just a doc patch for tempest-lib, but I think it'll be good to make sure I understood the restclient correctly :)
23:00:03 <mtreinish> ok if there aren't any other reviews we'll end here
23:00:09 <mtreinish> thanks everyone
23:00:26 <mtreinish> #endmeeting