17:00:13 <oomichi> #startmeeting qa
17:00:14 <openstack> Meeting started Thu Oct 13 17:00:13 2016 UTC and is due to finish in 60 minutes.  The chair is oomichi. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:18 <openstack> The meeting name has been set to 'qa'
17:00:30 <oomichi> hello, who are here today?
17:00:45 <DavidPurcell> o/ hello
17:01:03 <rodrigods> o/
17:01:22 <mtreinish> o/
17:01:52 <oomichi> thanks for attending. OK, let's start meeting
17:01:56 <oomichi> #link #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_October_13th_2016_.281700_UTC.29
17:02:06 <oomichi> ^^^ is today agenda
17:02:15 <oomichi> #topic Next Summit
17:02:28 <oomichi> #link https://etherpad.openstack.org/p/ocata-qa-summit-topics
17:02:44 <oomichi> We wrote some ideas for the summit on ^^^
17:02:55 <oomichi> and we need to select 7 slots for that
17:03:17 <oomichi> I arranged the order of proposals for my feeling
17:03:51 <oomichi> I feel top 3 seems good for discussion on the summit.
17:03:58 <oomichi> 1) policy testing
17:04:06 <oomichi> 2) plugins: roadmap for ..
17:04:20 <oomichi> 3) What's missing in Tempest
17:04:55 <oomichi> and if adding os-faults under QA, that also seems good to discuss at the summit
17:05:18 <oomichi> Do we have more topics needed to be added ?
17:05:55 <oomichi> or do you have some session needed to be discussed strongly?
17:07:34 <oomichi> ok, it will be nice to discuss more later at this time.
17:07:42 <oomichi> #topic Specs reviews
17:08:04 <mtreinish> oomichi: there is a ml thread about a devstack plugin for extra packages that might be a good topic too
17:08:17 <mtreinish> oomichi: you probably should talk to tonyb about it
17:08:58 <oomichi> mtreinish: thanks,, that is "devstack-plugin additional-pkg-repos: ocata summit working session?"
17:09:08 <mtreinish> oomichi: yeah
17:09:29 <oomichi> ok, I will catch tonyb
17:09:40 <mtreinish> it's not a qa project yet (although it probably would be a good one) but I think that might be a good use of a session slot
17:10:22 <oomichi> I put it on the etherpad
17:10:30 <oomichi> at line 39
17:11:05 <oomichi> mtreinish: thanks for bring it up
17:11:12 <oomichi> going back to the spec
17:11:29 <mtreinish> oomichi: it's probably worth just chiming in on the ml thread
17:12:02 <oomichi> mtreinish: yeah, will reply on the ml
17:12:35 <oomichi> but I need to read the thread much before that ;)
17:13:12 <oomichi> we have a new qa-spec #link https://review.openstack.org/#/c/382672/
17:13:40 <oomichi> I heard this kind of test requirement before
17:14:34 <oomichi> RBAC policy testing seems for developers and operators
17:14:46 <rj2083> in particular large operators
17:14:49 <rj2083> :)
17:15:02 <oomichi> rj2083: hehe, totally
17:15:33 <rj2083> it's really useful for large ops teams to be able to granularly add more roles, but sometimes the way the code is structured in the services, the policy.jsons lie
17:15:35 <oomichi> rj2083: yeah, I heard this kind of request from big operators
17:16:21 <mtreinish> so my issue with this spec (which I -1d on in the previous rev) is that this is scope creep if we add testing to tempest directly
17:16:23 <oomichi> rj2083: they just want to verify the policy.json works fine as they expected?
17:17:10 <rj2083> oomichi: basically yes, and if changing the policy.json for one role breaks something for another
17:17:17 <mtreinish> I think it's totally fair to add util functions to tempest.lib to make writing rbac tests using other tempest.lib pieces like the clients and cred providers (in the near future) easier
17:17:45 <mtreinish> but we don't want tempest owning any test code to handle an rbac scenario because that is inherently not compatible across clouds
17:18:00 <oomichi> mtreinish: yeah, good point. it is nice to reuse these library of tempest
17:18:16 <rj2083> in the case of the admin roles, the permissions in each service, even the actions as defiend in the policy.jsons don't actually work in some cases
17:18:16 <mtreinish> which is the dicussion we had before when ryan was first starting the cinnamon role work
17:18:28 <rj2083> i agree 100%, the tempest code should be re-usable
17:18:58 <rj2083> but i think security and rbac can't be an after thought and tempest tests should be integrated with rbac in a loosely coupled fashion
17:19:48 <mtreinish> rj2083: there is no such thing as loosely coupled. If the test code lives in tempest we have to own it which means running it
17:20:19 <mtreinish> and custom policy is inherently something we can't do that on, because it's a custom deployment change
17:20:25 <rj2083> i mean loosley coupled in the sense of not dictating a specific policy.json implementation tightly
17:20:45 <mtreinish> rj2083: well where tempest does that it's a community interop thing
17:21:04 <mtreinish> those are teh definitions of minimum things to maintain interop between clouds
17:21:42 <mtreinish> if deployers want to configure things to make it more involved than that it's fine, but if that causes tempest to break, then you've broken end user expecations
17:22:14 <rj2083> sure, but i think by limiting the flexibiltiy of the framework to two specific roles, that inherently limits the usefulness of tempest
17:22:15 <oomichi> yeah, deployers can custome policy.json as they want because of just configuration file
17:22:36 <oomichi> it is nice to keep tempest minimum on the policy testing
17:22:42 <rj2083> oomichi: for some roles yes, for admin roles, in most services the answer is no
17:23:10 <rj2083> but i think that clouds the issue
17:23:17 <rj2083> the spec in itself wouldn't really break tempest
17:23:36 <oomichi> rj2083: some normal users APIs are not available on some clouds and they want to change policy.json to controll these APIs from users
17:23:37 <mtreinish> rj2083: it's more than just 2 roles. But, that's what I'm saying is this is the bare minimum definition needed to work across all clouds
17:23:40 <rj2083> and i think adding the ability for large operators to test their roles doesnt inherently break end user expecations
17:23:55 <mtreinish> rj2083: no but you're asking for tempest to own a new type of testing which we explicitly don't want it to be in
17:24:06 <mtreinish> rj2083: I'm fine with the spec when it's just adding interfaces and utils to tempest.lib
17:24:20 <rj2083> ahh i see
17:24:38 <mtreinish> but when it starts discussing adding sample tests and new classes of testing to tempest (not lib) that's outside the scope of what tempest is
17:24:55 <rj2083> i can agree to that
17:24:56 <mtreinish> this is the whole reason cinnamon role was implemented as a plugin
17:25:43 <rj2083> in some cases however, the tempest tests do multiple policy actions at once and it's not inherently overridable by a decorator
17:26:23 <oomichi> rj2083: yeah, that is a nice point. some negative tests do that directly
17:26:48 <rj2083> oomichi: that's the basic problem we hit
17:27:07 <rj2083> the approach itself and even some of the positive tests did stuff like that
17:27:33 <oomichi> rj2083: how about just skipping such tests on running Tempest ;)
17:27:57 <rj2083> hahah that's one way  :) but our customers don't like that
17:28:12 <rj2083> and we'd prefer to use the upstream tests as closely as possible
17:28:15 <mtreinish> rj2083: do you have an example to link to?
17:28:17 <oomichi> rj2083: ok, they are strict on QA
17:28:55 <rj2083> it's not perfect, pretty rough, please be gentle :) https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/api
17:29:18 <mtreinish> because there should be sufficient wiggle room in tempest config to enable using different roles but still conform to the basic interop rules we're trying to enforce
17:29:22 <rj2083> https://github.com/DavidPurcellATT/TempestRbacDemo/blob/master/tempest/api/compute/rbac/test_services_rbac.py
17:29:36 <rj2083> mtreinish: that's 100% all i need
17:29:37 <rj2083> :)
17:29:44 <mtreinish> and if there isn't it might be your rbac rules are too restrictive
17:29:49 <rj2083> that would make me the happiest person in the world
17:29:51 <oomichi> #link https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/api
17:29:54 <rj2083> that and a puppy
17:30:21 <rj2083> oh my apoligies on the tags i'm using as i put this in the chat
17:30:25 <rj2083> #link https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/common/rbac
17:30:42 <rj2083> ^^ that's the framework we modified
17:30:48 <mtreinish> rj2083: no I'm saying there are config options to tell tempest to use specific roles for action types already
17:31:28 <DavidPurcell> It is a very rough version of our RBAC testing  framework that I ported to master (was written for kilo originally).  Very much a WIP.
17:31:29 <mtreinish> which should enable you to run the tests with more finely grained rbac
17:31:37 <rj2083> yes... we tried that, but some of the setup requires pre existing metadata that you can't create as the role you put in the settings
17:32:19 <mtreinish> rj2083: that's why I was asking for the example test (in tempest) you're having an issue with
17:32:26 <oomichi> mtreinish: I guess tempest doesn't have such config options to controll policy rules
17:32:27 <mtreinish> but we can discuss this after the meeting
17:32:39 <rj2083> yep that's cool :)
17:32:41 <oomichi> that is hard-coded in tests
17:32:49 <oomichi> yeah, that is a nice because of the time
17:32:56 <mtreinish> oomichi: it doesn't because we don't want to test policy since it's always custom
17:33:22 <mtreinish> oomichi: we have a couple predefined user types which are there for interop user type separation
17:33:25 <oomichi> mtreinish: yeah, and that should be minimum on tests
17:33:29 <rj2083> i know that's a general good guideline but there's tests or lines of openstack code where  where stuff is hardcoded to a role
17:33:35 <mtreinish> and the roles used for those are customizable
17:33:45 <mtreinish> but anything more fine grained or specific tempest shouldn't be directly testing
17:34:21 <oomichi> ok, let's move on the next topic if we can
17:34:27 <rj2083> k
17:34:41 <oomichi> #topic Tempest
17:34:55 <oomichi> I feel we have already much discussions about that ;)
17:34:59 <rj2083> hahah
17:35:06 <DavidPurcell> haha
17:35:13 <oomichi> does someone have topics about Tempest today?
17:35:31 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:lib-tenant-isolation
17:35:43 <mtreinish> that's my working branch to add dynamic creds to lib
17:36:02 <oomichi> mtreinish: oh, that all is green now. nice work :)
17:36:18 <mtreinish> it's not done yet, I still need to figure out the best final interface for it in lib
17:36:43 <mtreinish> like andreaf and I were discussing trying to split out the network allocation
17:36:55 <oomichi> mtreinish: then, that means you don't push the final patch for the pupose yet?
17:36:59 <mtreinish> but the patches up now are some necessary groundwork to cleanup some of the config usage, etc
17:37:18 <mtreinish> oomichi: I still need to cleanup the client manager usage (since it depends on non-lib code right now)
17:37:23 <mtreinish> and then all the prereqs are done
17:37:33 <mtreinish> but I might want to change the interface to make it a bit cleaner for lib
17:37:34 <oomichi> yeah, most patches are not difficult and make the code clean and easy
17:38:03 <oomichi> I prefer that
17:38:59 <oomichi> ok, do someone have another topic about Tempest
17:39:23 <mtreinish> #link https://review.openstack.org/#/c/371612/
17:39:33 <mtreinish> that's a good deprecation to land before the liberty eol
17:39:49 <oomichi> mtreinish: when is the liberty eol?
17:40:22 <oomichi> ah, 2016-11-17
17:40:30 <oomichi> according to #link https://releases.openstack.org/
17:41:02 <oomichi> ok, I will see it before that
17:41:23 <tosky> do you plan major tags before liberty-eol, or immediately after?
17:41:35 <oomichi> mtreinish: +2
17:42:14 <oomichi> tosky: we put major tags on each end of cycle(recently, newton)
17:42:23 <oomichi> not eol
17:42:28 <mtreinish> tosky: there will be a tag to mark the liberty eol
17:42:37 <mtreinish> oomichi: no we tag eol too
17:42:50 <oomichi> mtreinish: on master Tempest?
17:42:53 <mtreinish> oomichi: yes
17:43:05 <mtreinish> tosky: there might be tags between now and then, but that depends on whether we land anything worth pushing it
17:43:10 <mtreinish> oomichi: look at the git history
17:43:12 <tosky> ack, thanks
17:43:25 <oomichi> mtreinish: yeah, I should. sorry for making confusions
17:43:25 <mtreinish> oomichi: or read the docs: http://docs.openstack.org/developer/tempest/overview.html#release-versioning
17:44:20 <oomichi> mtreinish: thanks, that means we need to put 2 tags on each release, right?
17:44:34 <oomichi> s/tags/major tags/
17:44:55 <mtreinish> oomichi: I'm not following your question
17:45:11 <mtreinish> oomichi: we push one tag to mark the start of a release, and a 2nd to mark when it goes eol
17:45:34 <oomichi> mtreinish: yeah, that is I want to know
17:45:56 <mtreinish> that ends up being about 2 tags every 6 months (but that's assuming the same stable branch life for all branches)
17:46:33 <oomichi> yes, I know that today :)
17:46:51 <oomichi> ok, do we have more topics about Tempest?
17:47:09 <oomichi> or lets move on the next topic
17:47:19 <oomichi> #topic devstack + grenade
17:47:38 <oomichi> do we have more topics about them?
17:48:04 <oomichi> #topic openstack-health
17:48:17 <oomichi> do we have more topics about o-h today?
17:48:19 <timothyb89> the canvas charts should be ready for review: https://review.openstack.org/#/c/363934/9 https://review.openstack.org/#/c/380884/1 https://review.openstack.org/#/c/380885/1
17:48:25 <timothyb89> full example is available at http://logs.openstack.org/85/380885/1/check/gate-openstack-health-nodejs4-npm-run-test/9d90c2d/reports/build/#/
17:48:34 <mtreinish> timothyb89: ok, cool I'll try to take a look soon
17:49:04 <oomichi> good, thanks for bring them up
17:49:12 <oomichi> ok, lets move on
17:49:42 <oomichi> #topic critical review
17:49:54 <oomichi> #link https://review.openstack.org/#/c/363934
17:50:07 <oomichi> do we have more patches?
17:50:17 <oomichi> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:lib-tenant-isolation
17:51:18 <oomichi> #topic Open Discussion
17:51:57 <oomichi> I'd like to put some topic for that.
17:52:17 <oomichi> now we are seeing some new proposals related to QA
17:52:28 <oomichi> like os-faults, policy testing or something
17:52:55 <oomichi> it is nice to discuss them on qa-specs before adding, I feel
17:53:21 <oomichi> about their scopes and what difference from the existing projects
17:53:37 <oomichi> do we have some ideas or opinions about that?
17:54:19 <oomichi> in os-faults case, it just requires on -dev ml and gerrit
17:55:26 <oomichi> if no problem, I will discuss on -dev ml later.
17:55:35 <oomichi> do we have more topics today?
17:56:41 <oomichi> ok, lets end this meeting today. Thank you all :)
17:56:44 <oomichi> #endmeeting