17:00:55 <oomichi> #startmeeting qa
17:00:56 <openstack> Meeting started Thu Sep 29 17:00:55 2016 UTC and is due to finish in 60 minutes.  The chair is oomichi. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:59 <openstack> The meeting name has been set to 'qa'
17:01:10 <oomichi> hi, who are joining today?
17:01:20 <dustins> \o
17:01:22 <scottda> Hi
17:01:29 <mtreinish> o/
17:02:11 <oomichi> ok, lets start
17:02:21 <oomichi> #topic Next Summit
17:02:38 <oomichi> we have 7 slots for qa project design sessions.
17:02:58 <oomichi> and we are gathering some ideas for these slots on #link https://etherpad.openstack.org/p/ocata-qa-summit-topics
17:03:27 <oomichi> oops, I needed to pick up today agenda before that: #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_September_29th_2016_.281700_UTC.29
17:03:45 <oomichi> (going back to the summit)
17:04:20 <oomichi> now there is not good enough ideas and it is great to put the ideas on that more :)
17:04:45 <oomichi> rough ideas also are enough at this time
17:05:25 <oomichi> somebody has some ideas at this time?
17:06:06 <oomichi> ok, lets move to the next topic.
17:06:18 <oomichi> #topic Specs Reviews
17:06:36 <oomichi> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:07:00 <oomichi> there are not any changes since the previous meeting
17:07:36 <oomichi> on the spec. so I feel it is fine to skip this today if no topic on us
17:08:25 <oomichi> #topic Tempest
17:08:53 <oomichi> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
17:09:22 <oomichi> Now I am trying to define some guideline about negative tests on #link https://review.openstack.org/#/c/379013/
17:09:48 <oomichi> some good feedback is gotten already, so I will update it after this meeting
17:10:30 <oomichi> are there other items about Tempest today?
17:10:46 <DavidPurcell> I have something, if that's okay
17:11:01 <oomichi> DavidPurcell: Please go ahead :)
17:11:05 <DavidPurcell> Thanks
17:11:35 <DavidPurcell> The company I work for has written a framework and suite of tests for testing RBAC policy enforcement within Tempest.
17:11:53 <DavidPurcell> If there is interest, we were hoping to begin work on pushing it out to the community.
17:12:19 <mtreinish> DavidPurcell: that sounds similar to: https://review.openstack.org/#/c/314704/
17:12:40 <mtreinish> DavidPurcell: do you have a link to the tool?
17:12:46 <oomichi> DavidPurcell: that is interesting. Now which projects(nova, cinder, etct) are supported?
17:13:06 <DavidPurcell> Currently I think we have tests for: Nova, Glance, Cinder, Murano, Neutron, Heat
17:13:39 <DavidPurcell> I have seen the Cinnamon-Role project (the one in the review), but it seemed to be abandoned in May.
17:13:55 <oomichi> DavidPurcell: cool, yeah I also want to see the implementation way with some link if possible
17:14:32 <mtreinish> DavidPurcell: yeah I don't think rlossit has had time to touch it a while
17:14:43 <DavidPurcell> I didn't prepare something for this meeting, but I can have something to show soon
17:15:04 <oomichi> DavidPurcell: thanks :)
17:15:57 <DavidPurcell> mtreinish: Right, that is one of the reasons I was asking, I wanted to know if that project had been dropped, and if an alternative solution was open to consideration.
17:16:17 <oomichi> DavidPurcell: one question about this kind of tests: We can change policy on productions as they want, so how to verify these policies work fine on the tests?
17:16:53 <oomichi> The policy configuration files should be input on the tests?
17:17:00 <mtreinish> DavidPurcell: I think there would be, there has been desire for an rbac tool like this in qa. It's mostly been a matter of someone writing one
17:17:50 <mtreinish> the blocker on cinnamon-role was it's use of unstable tempest interfaces. We were planning to add it to qa when that was fixed (by stabilizing the interfaces into tempest/lib)
17:17:51 <DavidPurcell> Our current solution involves reading the apis from a yaml file we created based on the policies, and then using that to determine if the api being tested should be accessible by that role
17:18:49 <mtreinish> DavidPurcell: but if your implementation works, I definitely think it's worthwhile to look at it as an alternative
17:19:03 <mtreinish> it's hard to say anything concrete without having a repo to look at though
17:19:22 <oomichi> humm, interesting. yeah I am looking forward to seeing it
17:19:27 <DavidPurcell> Agreed, I wish I could just share it now, but internal restrictions :(
17:20:09 <DavidPurcell> Would the openstack-qa chat be the best place to discuss it once I have approval to share code?
17:20:10 <oomichi> DavidPurcell: no problem, that is common thing in companies
17:20:38 <oomichi> DavidPurcell: yeah, that is a right channel
17:20:51 <DavidPurcell> okay, thanks a lot!
17:21:23 <oomichi> DavidPurcell: thanks for picking this up. are there another topic about Tempest today?
17:21:37 <oomichi> all: are there another topic about Tempest today?
17:21:38 <mtreinish> I wanted to talk about this patch:
17:21:43 <mtreinish> #link https://review.openstack.org/#/c/373428/
17:22:04 <oomichi> mtreinish: ok, please go ahead :)
17:22:16 <mtreinish> it's still a wip, but I wanted to know if people had feelings on using ostestr as the underpinnings of tempest run instead of testr directly?
17:22:24 <mtreinish> because that's what the patch is proposing
17:23:05 <mtreinish> slowrie, dpaterson: ^^^
17:23:14 <oomichi> mtreinish: maybe do you have some concern about this?
17:23:44 <mtreinish> oomichi: well we made a decision to use testr directly when we did the initial implementation (I don't remember the reason why though)
17:23:45 <oomichi> mtreinish: the patch itself seems reducing the code from Tempest by using it
17:24:04 <mtreinish> it does but it leverages ostestr to do a lot of work under the covers
17:24:19 <mtreinish> and ostestr itself runs testr by subprocessing out
17:24:35 <mtreinish> which I think is my bigger concern, is that we're adding another fairly complex layer in the middle
17:24:53 <lhx_> oomichi, mtreinish, could you spare some minutes for review? https://review.openstack.org/#/c/378483/
17:25:42 <mtreinish> so the call path becomes tempest run -> ostestr -> testr -> CONC x subunit.run -> testtools.run -> unittest.run
17:26:16 <oomichi> mtreinish: oops, that seems very complex..
17:26:37 <mtreinish> oomichi: well even if you remove ostestr it's still complex :)
17:26:53 <mtreinish> I'm not opposed to doing this, but I wanted other people's thoughts on it
17:27:15 <oomichi> mtreinish: yeah, TBH I was surprised calls of testr at this time :)
17:27:42 <oomichi> mtreinish: ok, could you put some comment on https://review.openstack.org/#/c/373428 ?
17:27:54 <oomichi> mtreinish: the above explanation is good for reviewing
17:28:09 <oomichi> lhx_: sorry for late response, please go ahead
17:28:51 <oomichi> lhx_: ah, that is simple patch. I will review it after this meeting.
17:29:09 <lhx_> oomichi, sorry for interrupt you. thx for review
17:29:31 <oomichi> lhx_: no problem, and thanks for doing that
17:30:04 <lhx_> oomichi, haha, it's my job.
17:30:05 <oomichi> ok, are there any topic about tempest today?
17:30:19 <oomichi> lhx_: :)
17:30:55 <oomichi> ah, I need to bring one thing before moving
17:31:06 <oomichi> #link https://github.com/oomichi/bug-counter#current-graph
17:31:37 <oomichi> the bug number of Tempest is still decreasing by triage. Thanks for helping many people
17:31:52 <mtreinish> oomichi: hmm, we need to make the lines bigger in the legend. It's hard to see the colors
17:33:06 <mtreinish> oomichi: how many NEW bugs does tempest have? It's good to have that in single digits
17:33:14 <oomichi> mtreinish: hehe, yeah, it is great to do that. but that is not my work :)
17:33:18 <mtreinish> it looks like 30-40 by the graph
17:33:48 <oomichi> mtreinish: yeah, maybe around 30 on NEW
17:34:01 <oomichi> mtreinish: and the number seems increasing in this week
17:34:14 <oomichi> so we need to do some triage about that
17:34:56 <oomichi> mtreinish: do you have some idea for puting a digits on the graph?
17:35:47 <oomichi> and alternative way is #link https://review.openstack.org/#/c/372581/
17:35:57 <mtreinish> oomichi: that get's tricky to add, and makes it difficult to read. (that's why interactive graphs are nice) I can add more Y axis tick points to try and make it clearer
17:36:16 <oomichi> the patch adds long-term bug progress on bugstate
17:36:59 <oomichi> mtreinish: can we add Y axis on right side?
17:37:20 <mtreinish> oomichi: lets pick this up after the meeting
17:37:31 <oomichi> mtreinish: yeah, I agree
17:37:45 <oomichi> ok, let's move on
17:38:08 <oomichi> #topic DevStack + Grenade
17:38:38 <oomichi> Do we have some items necessary to discuss them today?
17:39:00 <mtreinish> the big thing for devstack and grenade this week is that we've branched them for newton
17:39:36 <oomichi> cool, we are moving to Ocata now
17:39:45 <mtreinish> looking at the newton branch though we still have some post branch stuff to do
17:40:18 <mtreinish> mostly just updating lib/tempest with the max microversions and the hard coded extension list
17:40:34 <mtreinish> we also need to do the extension list for liberty, and mitaka (that came up on the ML)
17:40:46 <mtreinish> oomichi: do you think you could push patches up for all that?
17:41:27 <oomichi> mtreinish: ah, I missed the mail of this topic maybe, is that for Nova extension or the other projects?
17:41:59 <mtreinish> #link http://lists.openstack.org/pipermail/openstack-dev/2016-September/104754.html
17:42:11 <mtreinish> it was about adding a neutron test for an extension added in mitaka
17:42:29 <mtreinish> we never hard coded the extension list on liberty or mitaka, so tests for new features fail on stable
17:44:21 <oomichi> mtreinish: humm, is it ok to just put some extensions list on stable devstacks?
17:45:10 <oomichi> I will follow that after meeting anyways.
17:45:29 <oomichi> #action oomichi follows http://lists.openstack.org/pipermail/openstack-dev/2016-September/104754.html
17:45:33 <mtreinish> it's a documented part of each release...
17:45:38 <mtreinish> #link https://wiki.openstack.org/wiki/QA/releases#Devstack
17:45:47 <mtreinish> it's part of branchless tempest's implementation
17:46:00 <mtreinish> but sure, we can follow up after the meeting
17:46:32 <oomichi> mtreinish: thanks. yeah it is nice to do that after meeting
17:46:56 <oomichi> do we have more topics about devstack + grenade today?
17:47:06 <oomichi> ok, lets move on
17:47:14 <oomichi> #topic openstack-health
17:47:36 <oomichi> #link https://review.openstack.org/#/q/project:openstack/openstack-health+status:open
17:48:24 <oomichi> do we have discussion items about o-h ?
17:48:39 <mtreinish> nothing from me this week
17:48:51 <oomichi> there are some patches on the list, and it is nice to just review
17:49:02 <oomichi> mtreinish: thanks, ok lets move on
17:49:17 <oomichi> #topic Critical Reviews
17:50:33 <oomichi> I'd like to pick #link https://review.openstack.org/#/c/379013/ up as a critical review. I hope a concenses is great from many reviewers
17:50:59 <oomichi> do we have more patches as critical or important?
17:51:52 <mtreinish> nothing from me this week
17:52:27 <oomichi> ok, let's move on
17:52:41 <oomichi> #topic Open Discussion
17:53:37 <oomichi> somebody have some discussion items today?
17:54:29 <oomichi> ok, lets end this meeting
17:54:33 <oomichi> thanks all
17:54:36 <oomichi> #endmeeting