09:00:21 <gmann_> #startmeeting qa
09:00:22 <openstack> Meeting started Thu Jul 28 09:00:21 2016 UTC and is due to finish in 60 minutes.  The chair is gmann_. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:25 <openstack> The meeting name has been set to 'qa'
09:00:36 <gmann_> hi, - who's here today?
09:00:39 <andreaf> o/
09:00:42 <dmellado> \o hi there
09:00:51 <vponomaryov> hello
09:01:10 <gmann_> hi everyone
09:01:38 <gmann_> let's wait if more people join
09:02:55 <andreaf> masayukig, jordanP, afazekas, mkoderer: around?
09:03:18 <gmann_> masayukig: is joining some openstack event today
09:03:29 <gmann_> anyways let's start
09:03:40 <gmann_> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_July_28th_2016_.280900_UTC.29
09:03:53 <gmann_> ^^ today agenda
09:04:13 <gmann_> #topic Newton priorities
09:04:24 <gmann_> #link https://etherpad.openstack.org/p/newton-qa-newton-priorities
09:05:06 <gmann_> so we have items there and some of  them in progress
09:05:37 <gmann_> i would like to avoid talking about each item
09:05:46 <gmann_> if anyone want to update on these
09:06:19 <gmann_> probably we are going in good shape on those.
09:06:49 <dmellado> I think so, althought for the test class hierarchy squash I'll wait for mid cycle
09:07:00 <gmann_> dmellado: ok
09:07:13 <gmann_> yea we can finish may of those there
09:07:43 <gmann_> if nothing let's move
09:07:49 <gmann_> #topic Specs Reviews
09:08:08 <gmann_> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
09:08:34 <gmann_> there no new spec
09:08:58 <andreaf> gmann_: indeed, probably nothing to discuss on this, unless someone as something to bring up
09:08:59 <gmann_> andreaf: tempest resource is on hold?
09:09:06 <gmann_> yea
09:09:17 <andreaf> gmann_: oomichi took that over, so I'm not looking at it right now
09:09:35 <gmann_> andreaf: ohk,
09:09:55 <gmann_> let's jump to tempest
09:10:02 <gmann_> #topic Tempest
09:10:32 <gmann_> 1 is about compute microversion tests
09:10:34 * masayukig is read only mode this time because he's on a train and walking to the event
09:10:53 <gmann_> Matt have doc update for those as per ML and nova mid cycle discussion
09:11:01 <gmann_> masayukig: +1
09:11:52 <gmann_> #link https://review.openstack.org/#/c/346092/
09:11:55 <gmann_> and nova one
09:12:04 <gmann_> #link https://review.openstack.org/#/c/346080/
09:12:26 <gmann_> but seems like we need to get conslusion about those in doc first
09:12:44 <gmann_> because there are some microvesion tests patches which may or may not be needed
09:13:25 <gmann_> so i brought this here to hold those microversion patches (touching Nova API/DB only ) till we have clear doc on thsoe
09:13:47 <gmann_> please provide your opinion on doc patches
09:13:56 <andreaf> gmann_: will do
09:14:03 <gmann_> andreaf: Thanks.
09:14:50 <vponomaryov> manila tempest plugin has support of microversions, if someone is interested may look at it
09:15:00 * afazekas hi
09:15:06 <gmann_> afazekas: hi
09:15:26 <jordanP> hi
09:15:31 <gmann_> vponomaryov: great. actually here question is do we need to implement tests for all microversion?
09:15:31 <andreaf> gmann_: my main concern is adding schema changes in tempest that are not tested - I agree we don't necessarily need to have a tempest test for each microversion, but we need some way to validate the schema, and that's not something we can do in unit tests unfortunately
09:15:35 <gmann_> jordanP: hi
09:15:36 <dmellado> hi afazekas jordanP
09:15:56 <vponomaryov> gmann_: we add tests for diff of each new microversion if possible
09:16:10 <gmann_> andreaf: definitely, we would not add schema which are not tested
09:16:33 <gmann_> andreaf: but if higher version tests cover those schema then it should be fine
09:16:58 <gmann_> as Nova always honor the lower version changes in higher one
09:17:31 <gmann_> vponomaryov: yea but sometime whose are duplicate with project functional tests
09:17:39 <gmann_> if the just change the API/DB layer only
09:17:42 <andreaf> gmann_: yes - the only thing we don't really test is that the lower version where we add the schema is the right one
09:17:59 <andreaf> gmann_: but that's probably not so relevant from a tempest pov
09:18:29 <gmann_> you mean to tests nothing more is returned in specific version
09:18:37 <gmann_> andreaf: yea that is nice point
09:19:21 <andreaf> gmann_: e.g. if we add a test for version 2.50 (just an example) and for that we add schema changes that came in in version 2.30, but we call those 2.31, it would work just fine, as long as there is no test for 2.30 or 2.31
09:19:36 <gmann_> I was thinking about filling the gap at end of cycle but from nova and tempest doc patch i feel we should test each microversion changing the schema
09:20:42 <gmann_> andreaf:  yea but without 2.30/31 tests we would not make sure it does not return more attributes than expected
09:22:26 <andreaf> gmann_: correct
09:22:40 <andreaf> gmann_: so are you saying we actually need a test for every microversion?
09:23:07 <gmann_> andreaf: every microversion which change the schema.
09:23:29 <gmann_> andreaf: is there is no schema change then first rule applicable (if touching only API/DB then no)
09:23:41 <gmann_> is->if
09:23:48 <andreaf> ok
09:24:35 <gmann_> actually that is current proposed in nova and tempest doc patches
09:24:39 <vponomaryov> all microversions change chema
09:24:54 <vponomaryov> they are added because of change of schema
09:25:01 <gmann_> vponomaryov: no, it depends on version to version
09:25:23 <gmann_> vponomaryov: no, there can be any kind of changes
09:25:43 <vponomaryov> gmann_: change of response code I consider as change of schema too
09:25:54 <gmann_> vponomaryov: for example, only request schema change or functional change or bug fix
09:26:13 <gmann_> vponomaryov: yea but not every microversion change response
09:26:37 <vponomaryov> it was example
09:26:59 <vponomaryov> microversions are added, when API started behaving differently
09:27:10 <gmann_> example  - https://github.com/openstack/nova/blob/master/nova/api/openstack/rest_api_version_history.rst#27
09:27:11 <vponomaryov> s/started/starts/
09:27:33 <gmann_> vponomaryov: yea which does not mean change in response only.
09:28:41 <gmann_> anyways let's review those and merged soon
09:29:28 <gmann_> next item i want to bring is very simple one
09:29:36 <gmann_> Removal of Javelin
09:29:37 <gmann_> #link https://review.openstack.org/#/c/347216/
09:29:45 <gmann_> will be pretty easy.
09:29:51 <dmellado> +1 from my side, we already marked it as deprecated a while ago
09:29:59 <jordanP> +1
09:30:35 <gmann_> I still need to create new repo for stress tests and thn remove from tempest
09:30:59 <gmann_> dmellado: yea its over 6 month
09:31:14 <gmann_> that'all from my side on Tempest
09:31:19 <andreaf> gmann_: is javelin going into it's own repo or just dropped?
09:31:28 <gmann_> andreaf:  just dropped
09:31:43 <gmann_> andreaf: i did not feel like anybody need so did not added new repo
09:31:51 <andreaf> gmann_: ok
09:32:09 <gmann_> andreaf: you want talk about client refactor one
09:32:09 <jordanP> yeah, just drop it
09:32:10 <andreaf> gmann_: are you looking at negative test framework as well?
09:32:34 <andreaf> gmann_: sure
09:32:38 <gmann_> andreaf:  no, may be in mid cycle we can discuss with mkoderer there :)
09:32:48 <andreaf> gmann_: ok
09:32:56 <andreaf> gmann_: regarding the client manager refactor
09:32:57 <gmann_> andreaf: i need to review as you updated those i think
09:33:23 <andreaf> gmann_: a few patches merged, but there are a few more https://review.openstack.org/#/q/status:open+branch:master+topic:bp/client-manager-refactor
09:33:59 <andreaf> gmann_: the series is ready from my perspective, I already got some +2 on some of the patches but it could use more attention
09:34:34 <gmann_> andreaf: may be this one and next one is key - https://review.openstack.org/#/c/333513/
09:34:44 <gmann_> andreaf:  after that it will be easy to merge others
09:35:26 <andreaf> gmann_: yes, that's probably the key one. I hope I addressed all the comments from you, mtreinish and oomichi on that one
09:35:48 <andreaf> gmann_: I changed it now to a fail fast behaviour in case of bad plugins
09:36:06 <gmann_> andreaf: Thanks, need to look on those. I will definitely do those 2 tomorrow.
09:36:12 <gmann_> and get back to you if anything
09:36:45 <andreaf> gmann_: since the service client interface is new, it's better not to be lenient on avoid broken implementations
09:36:59 <andreaf> gmann_: thanks
09:37:23 <dmellado> andreaf: so then once that this is done we'll need to address every plugin with patches like this one, isn't it?
09:37:24 <gmann_> ohk. i will check the updates
09:37:26 <dmellado> https://review.openstack.org/#/c/334596/
09:39:09 <andreaf> I have patches up for manila and monasca, I don't expect to do such patches for all plugins, but I will send an email to the ML and help people out where needed. I think a couple of examples should be sufficient for most teams to adapt their plugins
09:39:54 <gmann_> andreaf: +1, nice way. we cannot do for all plugin
09:40:07 <andreaf> gmann_: using the new interface should be quite straight-forward
09:40:14 <dmellado> andreaf: exactly, that was also my point
09:40:22 <gmann_> yea
09:40:26 <andreaf> gmann_: one limitation is that only stable clients are available
09:40:26 <dmellado> I'd be able to land a few too if needed, though
09:40:54 <andreaf> gmann_: which means that as long as volume, identity and object-storage are stable, they are not available to plugins via the stable interface
09:41:01 <gmann_> andreaf: yea, that is little nit tricky
09:41:25 <gmann_> andreaf:  user need to use both way if they want all clients
09:41:26 <andreaf> s/are stable/are not stable
09:41:52 <dmellado> maybe we'd need to get more help on getting them stable
09:41:54 <gmann_> but let's hope we will get those clients as stable soon.
09:42:10 <dmellado> AFAIK there are some projects, like manila who were complaning about that
09:42:30 <dmellado> vponomaryov: just pointing to that later, but is that part of your reason to skip some plugins?
09:42:54 <gmann_> dmellado: everyone complain on those when changed :)
09:43:05 <vponomaryov> dmellado: the reason is some other broken plugins were broken whole test run
09:43:08 <dmellado> gmann_: :D
09:43:19 <vponomaryov> s/were broken/were breaking/
09:43:23 <gmann_> ok let's move
09:43:28 <gmann_> anything else from anyone
09:43:51 <gmann_> vponomaryov: due to setup of plugins on gate as al plugin get setup there?
09:44:05 <vponomaryov> gmann_: yes, just by fact of their presence
09:44:19 <vponomaryov> that is unavoidable
09:44:30 <gmann_> vponomaryov: no, that is avoidable
09:44:44 <gmann_> vponomaryov: you can install only your plugin i think in venv
09:44:48 <gmann_> ironic did?
09:44:56 <gmann_> I need to check if those are got merged or not
09:45:06 <vponomaryov> gmann_: it is workaround
09:45:18 <gmann_> vponomaryov: yea,
09:45:19 <dmellado> vponomaryov: gmann_ but as they're know
09:45:23 <dmellado> and specifically to manila
09:45:31 <dmellado> as you point to an specific commit
09:45:34 <dmellado> if you install all plugins
09:45:37 <dmellado> things might break, imho
09:45:38 <dmellado> :\
09:45:59 <gmann_> yea
09:46:01 <vponomaryov> we used to breakage of tempest, sorry
09:46:21 <gmann_> humm
09:46:22 <gmann_> let's jump to critical review
09:46:23 <dmellado> I really do hope that finishing the stable clients would help on that
09:46:41 <gmann_> we can skip devstack and dashboard if nothing on those
09:46:45 <dmellado> +1
09:46:49 <andreaf> vponomaryov, dmellado, gmann_: the more stable interface we have the less breakages you may expect
09:46:49 <gmann_> #topic Critical Reviews
09:47:11 <gmann_> right
09:47:20 <andreaf> gmann_: #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/client-manager-refactor
09:47:23 <dmellado> andreaf: yep, that's why I'm trying to land a few patch series on those too
09:47:25 <gmann_> any reviews from anyone
09:47:33 <gmann_> andreaf:  noted :)
09:48:41 <gmann_> ok, we have 10 min left , let's move
09:48:44 <gmann_> #topic Possibility to enable/disable external plugins
09:48:49 <gmann_> vponomaryov: your one.
09:48:55 <vponomaryov> I proposed change (#link https://review.openstack.org/#/c/343830/), that allows us to enable/disable any external plugins we want just setting up one additional config option.
09:49:04 <vponomaryov> It has two pros:
09:49:09 <vponomaryov> - We can ignore undesired plugins
09:49:15 <vponomaryov> - We can avoid presence of tests from dozens of undesired projects in our generated test list.
09:49:29 <vponomaryov> So, I would like to get opinions on it.
09:50:09 <gmann_> vponomaryov: and it will do this while loading plugin?
09:50:15 <andreaf> vponomaryov: ignore tests is something you can already do via regex, just pick the tests from your plugin
09:50:35 <vponomaryov> gmann_: it allows us to influence loading of any plugin
09:50:50 <dmellado> vponomaryov: I assume you mean a plugin breaking your one, isn't it?
09:50:52 <andreaf> vponomaryov: I'm not against skipping plugins in general, but I think it's a bit of a workaround
09:51:06 <vponomaryov> andreaf: false, tests regex is far away of step when plugins are disabled
09:51:15 <andreaf> vponomaryov: and I fear it will reduce the pressure on broken plugins to be fixed
09:51:23 <gmann_> vponomaryov: then it is while loading the tests?
09:51:32 <vponomaryov> gmann_: loading plugins
09:51:37 <vponomaryov> gmann_: before it
09:51:43 <jordanP> why don't use just start only the tests you are interested in running, like: tox -e all-plugin -- manila.tests ?
09:51:45 <gmann_> vponomaryov: humm
09:51:53 <jordanP> ah ok, got it
09:52:01 <dmellado> also, what we were discussing before, wouldn't a solution -workaround too- have a venv containging only your desired plugins?
09:52:07 <dmellado> vponomaryov: ^^
09:52:09 <gmann_> jordanP: actually issue is loading of tests from other plugin if they import error
09:52:11 <vponomaryov> jordanP: because lots of redundant stuff is performed, if not broken at all
09:52:14 <jordanP> yeah
09:52:55 <vponomaryov> dmellado: I consider venv-based part as workaround
09:53:05 <andreaf> gmann_, jordanP: plugins should not do things at import time, same as tempest - so those should be fixed
09:53:05 <vponomaryov> dmellado: we should be able to use site-package-based plugins
09:53:06 <jordanP> I am not sure it's a good idea, broken plugins should be fixed or not installed at all...
09:53:29 <dmellado> jordanP: that might not be possible for in-tree packages
09:53:36 <andreaf> jordanP: yes that's my feeling too
09:53:38 <vponomaryov> jordanP: people install packages with some lugins, not just plugins
09:53:50 <jordanP> I think we wanted/want to move away from the proliferation of virtualenvs required to run tempest
09:53:56 <gmann_> jordanP: andreaf exactly. me too same feeling. they should be correct one or do not install
09:54:04 <dmellado> gmann_: but again
09:54:07 <vponomaryov> jordanP: I would like to not install some plugins of some packages, but it requires hacking only
09:54:08 <dmellado> how would you avoid having a plugin installed
09:54:13 <dmellado> if it comes with the in-tree project
09:54:16 <dmellado> such as the neutron one
09:54:18 <dmellado> ?
09:54:40 <vponomaryov> in proposed commit I just allow to ignore it in tempest
09:54:41 <dmellado> you might end up in a situation where if you don't use venv, you'll have quite a few plugins from start
09:54:42 <gmann_> dmellado: then do not install neutron.
09:54:43 <jordanP> if neutron's plugin is broken, then they have a big issue
09:54:54 <andreaf> dmellado: one more reason to have the plugin in a dedicated reop
09:55:01 <dmellado> andreaf: I do agree on that
09:55:02 <gmann_> venv things are very nice here.
09:55:06 <dmellado> and I'm trying to push it that way
09:55:09 <andreaf> s/reop/repo
09:55:24 <dmellado> but for instance I found some similar issue with the ironic plugin
09:55:30 <dmellado> which was even breaking testr list-tests
09:55:30 <gmann_> because tempest should load everything it detect
09:55:40 <dmellado> due to it trying to get credentials while it, which is quite weird
09:55:48 <vponomaryov> gmann_: we should be able to disable "load everything" that is the point
09:55:55 <dmellado> so I think that this could come in handy
09:56:16 <vponomaryov> gmann_: it is testing tool and shoudl be flexible enough
09:56:22 <vponomaryov> to serve lots of possible needs
09:56:26 <andreaf> dmellado, vponomaryov: plugins should not do things at import time, because that may break discovery
09:56:52 <dmellado> andreaf: I do agree on that, but I'm just thinking on a system when you'll end up not wanting to use venv
09:56:56 <gmann_> vponomaryov: but its all like my distribution. and what i installed should be in working condition instead of skiping those during runtime
09:57:00 <andreaf> vponomaryov: as I said I'm not opposed to the idea, but I'd like to make sure issues in plugins are fixed
09:57:00 <dmellado> while at the same time isolating/skipping plugins
09:57:17 <dmellado> andreaf: so you're fearing that with this, people would just don't care
09:57:22 <vponomaryov> andreaf: default behaviour is to load all
09:57:31 <vponomaryov> andreaf: "possibility" is added
09:57:50 <vponomaryov> andreaf: possibility to disable some
09:57:59 <vponomaryov> plugins
09:58:05 <dmellado> gmann_: I'll ask you about the way you handle plugins later, don't want to spam here
09:58:14 <gmann_> dmellado: sure
09:58:43 <jordanP_> what about surounding the plugin loading with a big try/except so that if a broken is broken at import/init, we just skip it ?
09:58:44 <gmann_> let's see the patch and get more feedback there
09:58:58 <gmann_> but i also still feel its all installation of system
09:59:03 <dmellado> jordanP_: that could be another option
09:59:14 <gmann_> jordanP_: might be not good, i tried this before :)
09:59:23 <dmellado> gmann_: and what happened?
09:59:29 <gmann_> jordanP_: it might hide lot of issues in plugin
09:59:31 <andreaf> jordanP_: sure that's an option, but that would make us even more lenient on broken plugins
09:59:40 <andreaf> gmann_ +1
09:59:41 <gmann_> yea
09:59:54 <dmellado> andreaf: then how do you propose to handle such plugins?
09:59:59 <jordanP_> broken plugins should be fixed by plugin owners, and not break Tempest
10:00:06 <jordanP_> time's up :)
10:00:15 <dmellado> just let tempest be broken (or use venv) so that'd force the devs to fix it?
10:00:20 <gmann_> jordanP_:  but they would get to know till tempest fail
10:00:29 <gmann_> anyways time is running i think
10:00:36 <dmellado> yeah, we're out of time
10:00:41 <dmellado> we can continue on #openstack-qa
10:00:46 <gmann_> let;s discuss this on QA channel or on review
10:01:10 <gmann_> Thanks  everyone
10:01:11 <gmann_> #endmeeting