17:00:13 #startmeeting qa 17:00:14 Meeting started Thu Oct 13 17:00:13 2016 UTC and is due to finish in 60 minutes. The chair is oomichi. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:18 The meeting name has been set to 'qa' 17:00:30 hello, who are here today? 17:00:45 o/ hello 17:01:03 o/ 17:01:22 o/ 17:01:52 thanks for attending. OK, let's start meeting 17:01:56 #link #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_October_13th_2016_.281700_UTC.29 17:02:06 ^^^ is today agenda 17:02:15 #topic Next Summit 17:02:28 #link https://etherpad.openstack.org/p/ocata-qa-summit-topics 17:02:44 We wrote some ideas for the summit on ^^^ 17:02:55 and we need to select 7 slots for that 17:03:17 I arranged the order of proposals for my feeling 17:03:51 I feel top 3 seems good for discussion on the summit. 17:03:58 1) policy testing 17:04:06 2) plugins: roadmap for .. 17:04:20 3) What's missing in Tempest 17:04:55 and if adding os-faults under QA, that also seems good to discuss at the summit 17:05:18 Do we have more topics needed to be added ? 17:05:55 or do you have some session needed to be discussed strongly? 17:07:34 ok, it will be nice to discuss more later at this time. 17:07:42 #topic Specs reviews 17:08:04 oomichi: there is a ml thread about a devstack plugin for extra packages that might be a good topic too 17:08:17 oomichi: you probably should talk to tonyb about it 17:08:58 mtreinish: thanks,, that is "devstack-plugin additional-pkg-repos: ocata summit working session?" 17:09:08 oomichi: yeah 17:09:29 ok, I will catch tonyb 17:09:40 it's not a qa project yet (although it probably would be a good one) but I think that might be a good use of a session slot 17:10:22 I put it on the etherpad 17:10:30 at line 39 17:11:05 mtreinish: thanks for bring it up 17:11:12 going back to the spec 17:11:29 oomichi: it's probably worth just chiming in on the ml thread 17:12:02 mtreinish: yeah, will reply on the ml 17:12:35 but I need to read the thread much before that ;) 17:13:12 we have a new qa-spec #link https://review.openstack.org/#/c/382672/ 17:13:40 I heard this kind of test requirement before 17:14:34 RBAC policy testing seems for developers and operators 17:14:46 in particular large operators 17:14:49 :) 17:15:02 rj2083: hehe, totally 17:15:33 it's really useful for large ops teams to be able to granularly add more roles, but sometimes the way the code is structured in the services, the policy.jsons lie 17:15:35 rj2083: yeah, I heard this kind of request from big operators 17:16:21 so my issue with this spec (which I -1d on in the previous rev) is that this is scope creep if we add testing to tempest directly 17:16:23 rj2083: they just want to verify the policy.json works fine as they expected? 17:17:10 oomichi: basically yes, and if changing the policy.json for one role breaks something for another 17:17:17 I think it's totally fair to add util functions to tempest.lib to make writing rbac tests using other tempest.lib pieces like the clients and cred providers (in the near future) easier 17:17:45 but we don't want tempest owning any test code to handle an rbac scenario because that is inherently not compatible across clouds 17:18:00 mtreinish: yeah, good point. it is nice to reuse these library of tempest 17:18:16 in the case of the admin roles, the permissions in each service, even the actions as defiend in the policy.jsons don't actually work in some cases 17:18:16 which is the dicussion we had before when ryan was first starting the cinnamon role work 17:18:28 i agree 100%, the tempest code should be re-usable 17:18:58 but i think security and rbac can't be an after thought and tempest tests should be integrated with rbac in a loosely coupled fashion 17:19:48 rj2083: there is no such thing as loosely coupled. If the test code lives in tempest we have to own it which means running it 17:20:19 and custom policy is inherently something we can't do that on, because it's a custom deployment change 17:20:25 i mean loosley coupled in the sense of not dictating a specific policy.json implementation tightly 17:20:45 rj2083: well where tempest does that it's a community interop thing 17:21:04 those are teh definitions of minimum things to maintain interop between clouds 17:21:42 if deployers want to configure things to make it more involved than that it's fine, but if that causes tempest to break, then you've broken end user expecations 17:22:14 sure, but i think by limiting the flexibiltiy of the framework to two specific roles, that inherently limits the usefulness of tempest 17:22:15 yeah, deployers can custome policy.json as they want because of just configuration file 17:22:36 it is nice to keep tempest minimum on the policy testing 17:22:42 oomichi: for some roles yes, for admin roles, in most services the answer is no 17:23:10 but i think that clouds the issue 17:23:17 the spec in itself wouldn't really break tempest 17:23:36 rj2083: some normal users APIs are not available on some clouds and they want to change policy.json to controll these APIs from users 17:23:37 rj2083: it's more than just 2 roles. But, that's what I'm saying is this is the bare minimum definition needed to work across all clouds 17:23:40 and i think adding the ability for large operators to test their roles doesnt inherently break end user expecations 17:23:55 rj2083: no but you're asking for tempest to own a new type of testing which we explicitly don't want it to be in 17:24:06 rj2083: I'm fine with the spec when it's just adding interfaces and utils to tempest.lib 17:24:20 ahh i see 17:24:38 but when it starts discussing adding sample tests and new classes of testing to tempest (not lib) that's outside the scope of what tempest is 17:24:55 i can agree to that 17:24:56 this is the whole reason cinnamon role was implemented as a plugin 17:25:43 in some cases however, the tempest tests do multiple policy actions at once and it's not inherently overridable by a decorator 17:26:23 rj2083: yeah, that is a nice point. some negative tests do that directly 17:26:48 oomichi: that's the basic problem we hit 17:27:07 the approach itself and even some of the positive tests did stuff like that 17:27:33 rj2083: how about just skipping such tests on running Tempest ;) 17:27:57 hahah that's one way :) but our customers don't like that 17:28:12 and we'd prefer to use the upstream tests as closely as possible 17:28:15 rj2083: do you have an example to link to? 17:28:17 rj2083: ok, they are strict on QA 17:28:55 it's not perfect, pretty rough, please be gentle :) https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/api 17:29:18 because there should be sufficient wiggle room in tempest config to enable using different roles but still conform to the basic interop rules we're trying to enforce 17:29:22 https://github.com/DavidPurcellATT/TempestRbacDemo/blob/master/tempest/api/compute/rbac/test_services_rbac.py 17:29:36 mtreinish: that's 100% all i need 17:29:37 :) 17:29:44 and if there isn't it might be your rbac rules are too restrictive 17:29:49 that would make me the happiest person in the world 17:29:51 #link https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/api 17:29:54 that and a puppy 17:30:21 oh my apoligies on the tags i'm using as i put this in the chat 17:30:25 #link https://github.com/DavidPurcellATT/TempestRbacDemo/tree/master/tempest/common/rbac 17:30:42 ^^ that's the framework we modified 17:30:48 rj2083: no I'm saying there are config options to tell tempest to use specific roles for action types already 17:31:28 It is a very rough version of our RBAC testing framework that I ported to master (was written for kilo originally). Very much a WIP. 17:31:29 which should enable you to run the tests with more finely grained rbac 17:31:37 yes... we tried that, but some of the setup requires pre existing metadata that you can't create as the role you put in the settings 17:32:19 rj2083: that's why I was asking for the example test (in tempest) you're having an issue with 17:32:26 mtreinish: I guess tempest doesn't have such config options to controll policy rules 17:32:27 but we can discuss this after the meeting 17:32:39 yep that's cool :) 17:32:41 that is hard-coded in tests 17:32:49 yeah, that is a nice because of the time 17:32:56 oomichi: it doesn't because we don't want to test policy since it's always custom 17:33:22 oomichi: we have a couple predefined user types which are there for interop user type separation 17:33:25 mtreinish: yeah, and that should be minimum on tests 17:33:29 i know that's a general good guideline but there's tests or lines of openstack code where where stuff is hardcoded to a role 17:33:35 and the roles used for those are customizable 17:33:45 but anything more fine grained or specific tempest shouldn't be directly testing 17:34:21 ok, let's move on the next topic if we can 17:34:27 k 17:34:41 #topic Tempest 17:34:55 I feel we have already much discussions about that ;) 17:34:59 hahah 17:35:06 haha 17:35:13 does someone have topics about Tempest today? 17:35:31 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:lib-tenant-isolation 17:35:43 that's my working branch to add dynamic creds to lib 17:36:02 mtreinish: oh, that all is green now. nice work :) 17:36:18 it's not done yet, I still need to figure out the best final interface for it in lib 17:36:43 like andreaf and I were discussing trying to split out the network allocation 17:36:55 mtreinish: then, that means you don't push the final patch for the pupose yet? 17:36:59 but the patches up now are some necessary groundwork to cleanup some of the config usage, etc 17:37:18 oomichi: I still need to cleanup the client manager usage (since it depends on non-lib code right now) 17:37:23 and then all the prereqs are done 17:37:33 but I might want to change the interface to make it a bit cleaner for lib 17:37:34 yeah, most patches are not difficult and make the code clean and easy 17:38:03 I prefer that 17:38:59 ok, do someone have another topic about Tempest 17:39:23 #link https://review.openstack.org/#/c/371612/ 17:39:33 that's a good deprecation to land before the liberty eol 17:39:49 mtreinish: when is the liberty eol? 17:40:22 ah, 2016-11-17 17:40:30 according to #link https://releases.openstack.org/ 17:41:02 ok, I will see it before that 17:41:23 do you plan major tags before liberty-eol, or immediately after? 17:41:35 mtreinish: +2 17:42:14 tosky: we put major tags on each end of cycle(recently, newton) 17:42:23 not eol 17:42:28 tosky: there will be a tag to mark the liberty eol 17:42:37 oomichi: no we tag eol too 17:42:50 mtreinish: on master Tempest? 17:42:53 oomichi: yes 17:43:05 tosky: there might be tags between now and then, but that depends on whether we land anything worth pushing it 17:43:10 oomichi: look at the git history 17:43:12 ack, thanks 17:43:25 mtreinish: yeah, I should. sorry for making confusions 17:43:25 oomichi: or read the docs: http://docs.openstack.org/developer/tempest/overview.html#release-versioning 17:44:20 mtreinish: thanks, that means we need to put 2 tags on each release, right? 17:44:34 s/tags/major tags/ 17:44:55 oomichi: I'm not following your question 17:45:11 oomichi: we push one tag to mark the start of a release, and a 2nd to mark when it goes eol 17:45:34 mtreinish: yeah, that is I want to know 17:45:56 that ends up being about 2 tags every 6 months (but that's assuming the same stable branch life for all branches) 17:46:33 yes, I know that today :) 17:46:51 ok, do we have more topics about Tempest? 17:47:09 or lets move on the next topic 17:47:19 #topic devstack + grenade 17:47:38 do we have more topics about them? 17:48:04 #topic openstack-health 17:48:17 do we have more topics about o-h today? 17:48:19 the canvas charts should be ready for review: https://review.openstack.org/#/c/363934/9 https://review.openstack.org/#/c/380884/1 https://review.openstack.org/#/c/380885/1 17:48:25 full example is available at http://logs.openstack.org/85/380885/1/check/gate-openstack-health-nodejs4-npm-run-test/9d90c2d/reports/build/#/ 17:48:34 timothyb89: ok, cool I'll try to take a look soon 17:49:04 good, thanks for bring them up 17:49:12 ok, lets move on 17:49:42 #topic critical review 17:49:54 #link https://review.openstack.org/#/c/363934 17:50:07 do we have more patches? 17:50:17 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:lib-tenant-isolation 17:51:18 #topic Open Discussion 17:51:57 I'd like to put some topic for that. 17:52:17 now we are seeing some new proposals related to QA 17:52:28 like os-faults, policy testing or something 17:52:55 it is nice to discuss them on qa-specs before adding, I feel 17:53:21 about their scopes and what difference from the existing projects 17:53:37 do we have some ideas or opinions about that? 17:54:19 in os-faults case, it just requires on -dev ml and gerrit 17:55:26 if no problem, I will discuss on -dev ml later. 17:55:35 do we have more topics today? 17:56:41 ok, lets end this meeting today. Thank you all :) 17:56:44 #endmeeting