17:01:17 <oomichi> #startmeeting qa
17:01:18 <openstack> Meeting started Thu Mar 17 17:01:17 2016 UTC and is due to finish in 60 minutes.  The chair is oomichi. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:22 <openstack> The meeting name has been set to 'qa'
17:01:34 <oomichi> hi, hi who's here for today?
17:01:39 <dmellado> o/
17:01:41 <raildo> o/
17:02:03 <jordanP> hi
17:02:13 <oomichi> andreaf, sdague, dtroyer, afazekas: around?
17:02:34 <rodrigods> o/
17:02:47 <oomichi> ok, well lets go through the agenda
17:02:55 <dpaterson> o/
17:03:08 <oomichi> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_March_17th_2016_.281700_UTC.29
17:03:16 <oomichi> ^^^ is today agenda
17:03:40 <oomichi> #topic Specs Reviews
17:04:00 <oomichi> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:04:09 <hogepodge> o/
17:04:18 <oomichi> most active specs come from andreaf
17:04:29 <oomichi> and the other specs are not so active now
17:04:56 <dmellado> yep, I owe you some reviews, will try to have that done tomorrow ;)
17:05:01 <oomichi> andreaf's ones seem good for me, and I will check the other ones later for cleanup
17:05:17 <oomichi> dmellado: cool, thanks in advance :-)
17:05:19 <jordanP> "Add bp tempest-resources" has not been updated for 3 weekens now
17:05:34 <jordanP> there's a couple of minor comments to address
17:05:39 <oomichi> jordanP: ah, nice point. but I prefer it :\;)
17:06:09 <oomichi> jordanP: ok, will catch andreaf later if we can update for merging
17:06:33 <oomichi> #action oomichi will catch andreaf for updating tempest-resources bp
17:07:09 <oomichi> now it is nice time to prepare austin summit, so please propose your idea as spec if having
17:07:45 <oomichi> does anyone have anything on this topic?
17:08:19 <oomichi> ok, then lets move on
17:08:36 <oomichi> #topic Priority Items
17:08:52 <oomichi> #link https://etherpad.openstack.org/p/mitaka-qa-priorities
17:09:15 <oomichi> that is the summary of previous summit
17:09:28 <jordanP> topic #10 is "Finalize ssh-auth bp" I think it's an interesting one and we shouldn"t drop it
17:09:41 <jordanP> hum... jlanoux is not here
17:09:51 <oomichi> jordanP: ok, do you know current status of that?
17:09:54 <dmellado> do you think that it'd be on time for mitaka? maybe we should postpone to N1
17:10:10 <jordanP> problem is, I don"t know what's missing
17:10:27 <oomichi> jordanP: heh
17:10:38 <jordanP> I think/thought, the overall goal is to enable ssh_validation in the gate
17:11:02 <jordanP> at the moment, ~6/7 tests are skipped because ssh validation is False in the gate
17:11:36 <oomichi> jordanP: do we have experimental jobs for enabling that?
17:12:05 <dmellado> if not, having one coupled with the changes would be great
17:12:14 <oomichi> it is nice to check current situation by enabling them temporary
17:12:21 <jordanP> yes one experiemental job is configured with ssh_validation = True
17:12:33 <ylobankov> we have the following job gate-tempest-dsvm-neutron-full-ssh
17:12:38 <oomichi> ah, cool
17:12:55 <oomichi> ylobankov: ah, I see.
17:12:58 <ylobankov> As far as I know all ssh related tests are enabled there
17:13:26 <jordanP> yes, that would be great to know how ofter these tests fail
17:13:41 <jordanP> anyway, let's ping jlanoux and/or andreaf when they are back
17:13:46 <dmellado> in any case we'll need jflanoux feedback
17:13:52 <oomichi> yeah, I tried to add some debugging log for ssh thing, but andreaf -1d because we have the job ;)
17:14:14 <oomichi> jordanP: yeah, will do
17:14:47 <jordanP> maybe next step is to make that job non voting but that will use a lot of resources
17:14:47 <oomichi> do we have another items for this topic?
17:15:17 <oomichi> jordanP: what kind of resource?
17:15:31 <oomichi> does that come from andreaf's spec?
17:15:34 <dmellado> I guess he means in terms on infra ones
17:15:56 <jordanP> yes, jenkins slave
17:16:00 <jordanP> VMs
17:16:38 <oomichi> jordanP: I got it. yeas, it is necesary to make such job as non-voting
17:17:27 <oomichi> ok, lets move to the next topic
17:17:41 <oomichi> #topic Tempest
17:17:55 <oomichi> raildo: hi
17:18:01 <raildo> oomichi: hey
17:18:07 <oomichi> raildo: the first one is your turn :-)
17:18:21 <raildo> it will be quicly :)
17:18:22 <oomichi> can you introduce that?
17:18:24 <raildo> hi guys, on Mitaka, Keystone team decided to deprecate API v2.0,
17:18:26 <raildo> since the API v3 it's now the stable API version, this job gate-tempest-dsvm-neutron-identity-v3-only-full test this API v3 behavior in the tempest.
17:18:37 <raildo> we can see the current job behavior here: http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-identity-v3-only-full-master?groupKey=project&resolutionKey=hour&duration=P1M so, since this job are running for a while (more that 6 months) and looking in this current behavior, I’m here to propose this job to be voting and add it in the gate too.
17:19:25 <raildo> sorry, I'm working most of my time on Keystone, and here to ask for help/tips on this job that Impact our job on it
17:19:47 <oomichi> raildo: do you have any patches for qa or infra noww?
17:19:52 <oomichi> s/noww/now/
17:19:55 <raildo> oomichi: https://review.openstack.org/#/c/292894/
17:21:08 <oomichi> raildo: most this team members dont have +2 for infra patches
17:21:32 <dmellado> raildo: besides that, you'll need to update this patch, won't you?
17:21:56 <raildo> dmellado: yes, I just keep the patch wip, to wait for this meeting
17:22:16 <raildo> dmellado: following the Andras suggestion...
17:22:21 <raildo> Andreas*
17:22:21 <jordanP> I am fine with making it voting
17:22:46 <dmellado> same here, seems pretty stable and looks reasonable to me
17:22:53 <oomichi> yeah, the job seems stable now
17:23:08 <dmellado> raildo: could you update the patch as per Andreas comments? I'll review it afterwards
17:23:13 <raildo> great, so I'll fix the Andreas comments and send a new patch set today
17:23:25 <oomichi> raildo: cool, thanks in advance
17:23:34 <raildo> dmellado: sure, I'll ping you today :)
17:23:37 <raildo> thanks guys!
17:23:44 <oomichi> ok, the next is mine
17:23:47 <dmellado> np raildo ;)
17:24:06 <oomichi> I am proposing to remove negative tests from tempest now
17:24:24 <jordanP> raildo, but you should be aware that some project (Heat, Nova at least) can only deal with keystone v2
17:24:36 <jordanP> raildo, have a look at https://bugs.launchpad.net/python-novaclient/+bug/1522402
17:24:37 <openstack> Launchpad bug 1522402 in python-novaclient "Novaclient does not support v3 Keystone API" [High,In progress] - Assigned to Andrey Kurilin (andreykurilin)
17:24:44 <jordanP> Keystone is not done with v2 yet
17:25:12 <raildo> jordanP: yes, I'm aware on this too, we are working to fix this v2/v3 issues
17:25:13 <rodrigods> jordanP, hmm ppl need to hurry, v2 is under deprecation
17:25:18 <jordanP> making v3 job voting is great, but don"t forget that most users/projects are stuck with v2
17:25:28 <andreykurilin> jordanP: novaclient works with keystone v3, but it doesn't have several default values :)
17:25:32 <dstanek> oomichi: that sort of confused me. what is the mission for tempest exactly?
17:25:57 <oomichi> dstanek: that is integration tests, not corner case tests
17:26:01 <htruta> jordanP: we are working on improving this support for v3 in a lot of projects
17:26:01 <jordanP> rodrigods, ppl won't hurry. Heat and Nova can only speak v2 so Keystone should support v2 longer
17:26:19 <jordanP> htruta, that's good to know
17:26:23 <htruta> jordanP: v2 won't be removed before Q
17:26:37 <rodrigods> long deprecation period :P
17:26:53 <jordanP> ok, back to negative tests then
17:27:00 <oomichi> dstanek: most negative tests are corner case tests, I am fine to keep some negative tests which are not conner
17:27:00 <dstanek> oomichi: so no failure scenarios?
17:27:03 <htruta> jordanP: our next step is to create other gates (even as non-voting) in services like nova, to more accurately identify those issues
17:27:04 <jordanP> I thin they don"t hurt and some of them are useful
17:27:15 <rodrigods> ++
17:27:29 <oomichi> dstanek: sorry, what does mean that?
17:27:34 <dmellado> oomichi: about that I was reading your email and agree on jordanP comments, what would you expect to gain removing them?
17:28:04 <oomichi> dmellado: nice point.
17:28:12 <rodrigods> services aren't ready yet to host these kinds of tests too
17:28:17 <oomichi> there are several merits
17:28:27 <dstanek> oomichi: for example, maybe in the middle of running an operation keystone isn't available and the service retries. (this is just a strawman)
17:28:44 <oomichi> one is to clarify what project problem on the gate by separating tests
17:28:57 <oomichi> with tempest plugin interface
17:29:04 <jordanP> oomichi, but the negative tests only take 150 seconds max
17:29:15 <jordanP> this is not where we should spend effort
17:29:36 <ylobankov> For me keeping negative tests in the Tempest tree is not so bad
17:29:39 <jordanP> we risk to introduce coverage gap, API regression just to save 150 seconds...
17:29:45 <oomichi> jordanP: 150 sec for all tests
17:29:59 <jordanP> yes, the 450 negatives tests we have
17:30:15 <oomichi> oh, nice point
17:30:34 <oomichi> then it is reasonable to keep them
17:30:38 <dmellado> also oomichi
17:30:46 <dmellado> you were mentioning about having those in each component's repo
17:30:47 <rodrigods> good
17:30:52 <jordanP> negative tests do catch some API consistency. For instance here: https://review.openstack.org/#/c/278383/
17:30:54 <dmellado> but not every component has alrady them
17:30:57 <oomichi> is it nice to add more negative tests later?
17:30:58 <rodrigods> jordanP, ++
17:31:01 <dmellado> s/alrady/already
17:31:18 <dmellado> so I think that maybe we could agree on new ones going to each component repo
17:31:26 <rodrigods> example of API break: https://review.openstack.org/#/c/292827/
17:31:28 <dmellado> but we'll maintain the ones that we have until them
17:31:37 <rodrigods> that would be difficult to have if we had this test before
17:31:50 <jordanP> well I don't think we should add more. The contrary, we should say no to negative tests by default, but if they are really useful (that's hard to judge, I know) then maybe accept them
17:31:51 <oomichi> jordanP: it is possible that even if implementing that with tempest plugin
17:32:07 <oomichi> jordanP: about https://review.openstack.org/#/c/278383/
17:32:20 <rodrigods> jordanP, oomichi, can accept while we don't have the infra in the services?
17:32:23 <jordanP> yes sure. Tempest plugins are great, projects should transition to that
17:32:32 <rodrigods> i mean, if the service can't run tests from the tempest plugin yet?
17:32:57 <jordanP> rodrigods, we need to start to say "no, please do that in your tempest-plugin"
17:33:13 <oomichi> rodrigods: what project?
17:33:19 <rodrigods> keystone, for instance
17:33:21 <jordanP> otherwise projects won't move/adopt the plugin model
17:33:30 <dmellado> in terms of that, and sorry for the offtopic, I'd appretiate reviews on the WIP for the neutron one: https://review.openstack.org/#/c/274023/
17:33:46 <rodrigods> jordanP, makes sense
17:34:00 <rodrigods> but... we would lose some tests during the process
17:34:04 <dstanek> jordanP: where can i find docs for making a tempest-plugin?
17:34:09 <oomichi> dmellado: ok, wil check it
17:34:13 <dmellado> dstanek:
17:34:19 <dmellado> it's on the tempest dev doc
17:34:22 * dmellado checking
17:34:27 <jordanP> http://docs.openstack.org/developer/tempest/plugin.html
17:34:33 <dmellado> there you are
17:34:39 <jordanP> https://opensource.com/business/15/10/creating-a-tempest-plugin-for-openstack
17:35:17 <dmellado> jordanP: so there's even an article about that... xD
17:35:28 <rodrigods> do you have cycles to help on that front dstanek?
17:35:37 <dstanek> rodrigods: yes
17:35:38 <jordanP> I will start to softly -1 patches that add long running tests and that aren"t integration/multi project tests
17:35:51 <dmellado> but what about the scenario tests
17:36:01 <dmellado> would it make sense to move them too, to the 'closest' project?
17:36:02 <rodrigods> dstanek, great, let's start to make this than
17:36:21 <rodrigods> dmellado, don't think the idea is to move
17:36:23 <dstanek> rodrigods: i'll read through the docs tonight
17:36:25 <rodrigods> but to not add
17:36:28 <jordanP> dmellado, same. Scenarios are great, we need more of them. But if someone wants to add a 3minutes scenarios that only use Neutron or only Cinder, I would say no
17:36:47 <dstanek> jordanP: thanks for the links
17:36:56 <rodrigods> dstanek, same here
17:36:56 <oomichi> jordanP: yeah, agree
17:37:16 <dmellado> jordanP: my concern on that is that we'd need to move some other stuff
17:37:18 <dstanek> has any project started creating tempest plugins for thier integration testing?
17:37:28 <jordanP> dteselkin, manila is a good example
17:37:30 <dmellado> such as creating servers
17:37:37 <dmellado> to the project 'scenario manager'
17:37:43 <oomichi> dstanek: ironice also is doing now
17:37:48 <dmellado> and that's when I think thins start getting blurry
17:37:53 <dmellado> would it be neutron? nova?
17:38:07 <dmellado> would neutron need to have nova methods in such 'manager'
17:38:09 <dmellado> ?
17:38:17 <oomichi> dmellado: not yet because they are core
17:38:35 <jordanP> they can use the current scenario manager
17:38:38 <oomichi> dmellado: on current rule, all tests of core projects are kept in tempest
17:38:48 <dstanek> dmellado: i imagine that this will be somewhat difficult for complicated scenarios
17:39:02 <dmellado> dstanek: that's my concern
17:39:15 <dmellado> oomichi: then why have we moved tests to neutron, for instance?
17:40:06 <oomichi> dmellado: if we need to implement more negatvie tests for them, that would be necessary
17:40:35 <rodrigods> jordanP, i'll be working on adding tempest plugins support in keystone, but seems ok to get this in https://review.openstack.org/#/c/294165/ ?
17:40:37 <dmellado> oomichi: I see
17:40:56 <dmellado> oomichi: but for instance let's say I'd like to move this scenario test to
17:40:58 <dmellado> https://review.openstack.org/#/c/217177/
17:41:01 <dmellado> neutron repo
17:41:11 <dmellado> as it uses a neutron feature
17:41:21 <dmellado> would the plugin handle that?
17:41:32 <dmellado> maybe I'm missing something on it but there's where I see blurryness ;)
17:41:42 <oomichi> dmellado: I am not sure at this time
17:41:50 <jordanP> me neither. But I would say yes
17:42:14 <jordanP> we shouldn't remove tests from Tempest. But new tests should be added to the projects' repo
17:42:26 <oomichi> dmellado: but I guess ironice scenario tests have been moved  to the repo, so it would be possible
17:42:41 <dmellado> oomichi: jordanP cool from now, I'll be helping draft this so I'll get back to you about the process, to see if we could draft some guidelines ;)
17:42:43 <oomichi> dmellado: I will check the code later
17:42:53 <dmellado> thanks oomichi ;)
17:43:15 <oomichi> dmellado: I want to help it for that :)
17:43:34 <dmellado> awesome!
17:43:58 <dmellado> should we move to next topic now?
17:44:03 <oomichi> are there any topic now?
17:44:13 <oomichi> ok, lets move on
17:44:15 <jordanP> wait, what did we agree about negative tests ?
17:44:20 <jordanP> :)
17:44:20 <dmellado> lol
17:44:25 <dmellado> we're keeping them, aren't we?
17:44:38 <jordanP> I'll wait for the future PTL to say :)
17:44:56 <oomichi> jordanP: oh, nice point. how about discussing more it on -dev?
17:45:02 <jordanP> again ? :)
17:45:08 <jordanP> I thought we had the discussion already
17:45:25 <oomichi> at least, more negative tests are recommended to add with tempest plugin
17:45:38 <jordanP> yeah sure
17:45:43 <jordanP> but what about existent ones ? :)
17:45:50 <hogepodge> defcore has negative tests as part of the standard, and at our last meeting we decided to not remove them from the standard
17:45:51 <dmellado> oomichi: jordanP from my point of view we should keep existing ones, and add next ones to their projects
17:46:04 <jordanP> dmellado, I agree with you
17:46:30 <oomichi> the existing ones attract more negative tests
17:46:31 <ylobankov> dmellado jordanP I agree with you
17:46:53 <jordanP> then let's say -2 to the new ones
17:47:00 <jordanP> "please do this in a plugin"
17:47:12 <oomichi> jordanP: ok, I agree with you
17:47:22 <dmellado> sounds like a deal then ;)
17:47:54 <oomichi> ok, let's move on
17:48:04 <jordanP> yes
17:48:10 <oomichi> #topic DevStack + Grenade
17:48:13 <dmellado> hogepodge: that would mean that maybe on the next refstack list the plugins will be have to be taking into consideration
17:48:32 <dmellado> s/taking/taken
17:48:35 <dmellado> ;)
17:48:51 <hogepodge> dmellado: we're already looking at plugins, and apparently advisory neutron api calls were moved out of tempest recently
17:49:04 <hogepodge> s/calls/tests/
17:49:07 <dmellado> hogepodge: exactly
17:50:24 <oomichi> any one have items about this topic?
17:50:38 <oomichi> about devstack and grenade
17:51:07 <oomichi> ok, lets move on
17:51:11 <oomichi> #topic Austin Summit
17:51:33 <oomichi> #link https://etherpad.openstack.org/p/newton-qa-summit-topics
17:51:52 <oomichi> the link is for getting ideas for austin summit :)
17:52:15 <oomichi> it is great to add more ideas on that ;)
17:52:39 <oomichi> QA team has 4fishballs and 4 working spaces
17:52:46 <oomichi> meaning 8 slots
17:53:01 <dmellado> oomichi: would those be for talks or just conversations/workshops?
17:53:39 <oomichi> dmellado: for discussion of next features
17:53:50 <dmellado> then let's add a tempest plugin one ;)
17:54:16 <oomichi> dmellado: "what is necessary for tempest plugin" or something?
17:54:34 <dmellado> sounds totally fine
17:54:46 <jordanP> tempest-plugin is almost a done deal. We need adoption know
17:54:50 <jordanP> *now
17:55:28 <oomichi> jordanP: yeah, but more readable doc or something is good for adoption.
17:55:37 <rodrigods> examples
17:55:43 <jordanP> and a reference implmentation
17:55:45 <jordanP> yeah examples
17:55:50 <dmellado> jordanP: I'm saying this as I'm getting ton of questions about that
17:55:52 <oomichi> yeah, nice
17:56:06 <dmellado> from other team's folks, who'd be the users and approvers in the end
17:56:18 <oomichi> dmellado: that seems nice, can you add it ?
17:56:22 <oomichi> on the etherpad
17:56:23 <dmellado> sure
17:56:28 <oomichi> dmellado: thanks
17:57:13 <oomichi> do anyone want to talk about the summit?
17:57:22 <oomichi> ok, lets move on
17:57:24 <dmellado> oomichi: about the bbq? ;)
17:57:39 <oomichi> dmellado: BBQ?
17:57:56 <dmellado> just joking, but we should organize something there ;)
17:58:10 <oomichi> dmellado: ah, hehe, I see ;)
17:58:24 <oomichi> austin is the best place, I guess ;)
17:58:45 <oomichi> #topic Critical Reviews
17:59:04 <ylobankov> https://review.openstack.org/#/c/275497/
17:59:05 <oomichi> are there any patches to be reviewed?
17:59:17 <oomichi> #link https://review.openstack.org/#/c/275497/
17:59:20 <ylobankov> not critical, but it is simple patch
17:59:51 <ylobankov> and nobody reviews it :(
18:00:03 <oomichi> ylobankov: I reviewed the similar patch..
18:00:06 <rodrigods> not critical too, i have some
18:00:06 <dmellado> ylobankov: added to my queue
18:00:17 * oomichi de-ja-bu
18:00:32 <oomichi> rodrigods: please :)
18:00:43 <oomichi> oh, sorry time comes
18:00:47 <oomichi> thanks all
18:00:51 <oomichi> #endmeeting