17:00:31 <andreaf> #startmeeting qa
17:00:32 <openstack> Meeting started Thu Dec 22 17:00:31 2016 UTC and is due to finish in 60 minutes.  The chair is andreaf. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:35 <openstack> The meeting name has been set to 'qa'
17:00:39 <jordanP> hi
17:00:52 <andreaf> hello dear QA engineers - who's here today?
17:00:56 <scottda> hi
17:00:57 <andreaf> hi jordanP :)
17:01:04 <andreaf> o/
17:01:21 <andreaf> Hello scottda
17:01:56 <andreaf> let's wait a couple of minutes more to see if someone else joins
17:02:12 * andreaf thinks the holiday season has started for many already
17:02:18 <jordanP> yup
17:02:35 <rodrigods> o/
17:02:45 <mmedvede> o/
17:02:53 <andreaf> Agenda for today: #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_December_22nd_2016_.281700_UTC.29
17:03:07 <andreaf> ok it's 3 past, let's get started
17:03:55 <andreaf> First topic, I just wanted to quickly remind folks about the PTG
17:04:03 <andreaf> #link https://www.openstack.org/ptg/
17:04:25 <andreaf> I don't think we have an etherpad with topics yet, but talking with folks a couple of things emerged
17:04:36 <andreaf> i.e. squash class hierarchy, fix neutron scenario
17:04:43 <jordanP> who will be there ? I will
17:04:49 <andreaf> which would be good to tackle
17:04:53 <andreaf> I will be there
17:05:10 <rodrigods> i won't :( will miss lots of keystone stuff too
17:05:17 <andreaf> I think oomichi will be there
17:05:19 <andreaf> :(
17:05:24 <DavidPurcell> I live in Atlanta, so I will be there.
17:05:33 <rodrigods> DavidPurcell, ++
17:06:23 <andreaf> so beginning of next year we should start an etherpad and start to decide in advance what we are going to be working on to make the best of our time together there
17:06:41 <jordanP> squash class hierarchy, fix neutron scenario are good topics btw
17:06:47 <andreaf> #action oomichi setup and etherpad for QA at the PTG
17:07:00 * andreaf assigns actions to absents :P
17:07:16 <jordanP> lol
17:07:18 <andreaf> ok anything else on PTG?
17:07:22 <rodrigods> lol
17:07:58 <andreaf> #topic Specs Reviews
17:08:05 <andreaf> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:08:18 <andreaf> anything on specs?
17:08:36 <rodrigods> andreaf, there is one ayoung wants to revert
17:08:51 <rodrigods> maybe we can discuss?
17:08:53 <ayoung> sure
17:08:55 <rodrigods> if he is available...
17:08:59 <rodrigods> oh there he is
17:09:05 <andreaf> link?
17:09:10 <ayoung> one sec
17:09:54 <andreaf> heh I guess #link https://review.openstack.org/#/c/410250/
17:10:01 <ayoung> having troubler getting to Gerrit...
17:10:08 <rodrigods> yeah... that one
17:10:09 <ayoung> Yeah...ok
17:10:12 <ayoung> so here is the deal
17:10:27 <ayoung> we've been associating policy and RBAC for so long, that you guys thought we were serious
17:10:32 <ayoung> we were, but we are not anymore
17:10:39 <ayoung> here is where we are headed:
17:10:59 <ayoung> we are planning on splitting RBAC from policy, and enforcing RBAC earlier, in the keystonemiddleware phase
17:11:10 <ayoung> there are many reasons for this, and you can read them on the spec I linked in the review...
17:11:19 <ayoung> http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/role-check-from-middleware.html
17:11:47 <ayoung> Since this is totally unfair to you guys, I figured I would give you a big heads up by proposing a Revert
17:11:54 <ayoung> :}
17:12:04 <ayoung> there are a couple things we could do
17:12:25 <ayoung> 1.  rewrite this spec to be a general test of the policy framework, with a focus on the scope checking:
17:12:45 <ayoung> scope checking meaning "does the project of the token match the project of the resource"
17:13:05 <ayoung> this is the main thing that we are going to expect oslo-policy based code to continie to enforce
17:13:18 <ayoung> there will also be service specific things...see the neutron policyu file for examples
17:13:50 <rj2083> wassup
17:13:53 <ayoung> the roles needed to execute a specific operation, however, won't be a part of that, and the reverted spec was targeting exactly that
17:14:19 <ayoung> so  the other thing we could do is
17:14:34 <ayoung> 2.  CHange this spec to test the RBAC as defined by the new mechanism
17:14:37 <rj2083> regardless of where you enforce rbac, you still need to test it
17:14:38 <ayoung> and that would be wonderful
17:14:45 <ayoung> rj2083, yep
17:15:03 <ayoung> and I am willing to drop the revert so long as we have a plan to test it as smart as possible
17:15:08 <DavidPurcell> I like the 2nd approach.
17:15:30 <ayoung> DavidPurcell, so do I.  The biggest issue is that the RBAC in middleware is still under development
17:15:35 <andreaf> would we be able to write tests now that will work once the new mechanism is in place?
17:15:41 <ayoung> but doing QA along side the main code would be wonderful
17:15:42 <rodrigods> 2nd approach makes sense, although this needs to land first in keystonemiddleware side
17:15:57 <ayoung> rodrigods, right...3 layers actually
17:16:00 <ayoung> 1.  keystone server
17:16:03 <ayoung> 2. keystoneclient
17:16:06 <ayoung> 3. keystonemiddleware
17:16:08 <rodrigods> yeah
17:16:11 <ayoung> I have  WIP for 1
17:16:20 <rj2083> and actually four layers
17:16:30 <DavidPurcell> ayoung: Right.  That is why I wanted to start off testing the current method that centers around policies, and then once the new approach is added to Keystone, we can consider adding or changing to that approach.
17:16:31 <rj2083> one for every service to pull out oslo.policy enforcement
17:16:42 <ayoung> DavidPurcell, so I think we really want to test both
17:16:53 <ayoung> when I said two options, I really mean, there are two things we need to do
17:17:06 <ayoung> 1 is to redirect the current effort to testing policy, but not focus on RBAC
17:17:09 <jordanP> (DavidPurcell, is that related to the Patrole project ?)
17:17:12 <ayoung> 2 is to test RBAC in the middleware layer
17:17:44 <rj2083> in actuality all of this is external to patrole
17:18:00 <ayoung> the example in the spec talks about testing RBAC for "compute_extension:services"
17:18:23 <ayoung> now, one major difference in the Middleware approach is that RBAC will be per URL, not per policy rule
17:18:25 <rj2083> i'm not sure why we would want to do any of this in your buleprint andrew, before the other services have removed oslo policy
17:18:33 <rj2083> and defined everything in keystone alone
17:18:36 <ayoung> so instead of  "compute_extension:services"
17:19:02 <ayoung> is would be service=compute url=/v2.1/thing/that/extends/services
17:19:04 <ayoung> or whatnot
17:19:43 <ayoung> I am really worried about Tempest and other tests locking us in to the existing policy approach based on RBAC, and then not being able to get the base code in
17:20:02 <ayoung> I've seen that happen on other policy based work, so I did want to run up a big RED FLAG
17:20:14 <ayoung> so, sorry for being a little grandiouse in the revert, but the message was important
17:20:15 <rj2083> until it happens here
17:20:21 <rj2083> i'm not worried
17:20:35 <rj2083> and actually regardless of where the enforcement happens
17:20:41 <rj2083> the api flow will remain unchanged
17:20:46 <jordanP> ayoung, I think you did right. We (QA) shouldn't write tests that will delay or block what projects want
17:20:51 <ayoung> thanks
17:20:58 <jordanP> I better have no test than wrong tests
17:21:06 <jordanP> that's just me though :)
17:21:12 <rj2083> you will always call a service and the service will always enforce policy
17:21:19 <ayoung> jordanP, and in this case, it will be a multi-project effort to get things set up.  I've been dealing with one of those for bug 968696
17:21:21 <openstack> bug 968696 in OpenStack Identity (keystone) ""admin"-ness not properly scoped" [High,In progress] https://launchpad.net/bugs/968696 - Assigned to Adam Young (ayoung)
17:21:24 <rj2083> regardless of what it defers to or calls ono the back end
17:21:30 <rj2083> none of the tests even need to be changed for this
17:21:58 <ayoung> rj2083, one of the big things is to defing the RBAC tests in terms of the APIs instead of the oslo-policy rules
17:22:06 <andreaf> jordanP, ayoung: well with Tempest we need to ensure backward compatibility on APIs, so if some API exists which is wrong today, we still need to ensure backward compatibility
17:22:12 <ayoung> the apis are almost all speced out in the api-ref docs...
17:22:36 <rj2083> that doesnt change the tests or how the spec is written though
17:22:41 <andreaf> jordanP, ayoung: that said I don't think this will have impact on the APIs or on the end user directly, willit?
17:22:53 <rj2083> and it doesnt invalidate or chanve any of the tests at all actualy
17:22:56 <ayoung> andreaf, it should not.  Just on how the tests are written
17:23:03 <ayoung> rj2083, it might
17:23:21 <ayoung> rj2083, there is the case where a single policy rule might be enforcing on two or more APIs
17:23:24 <rj2083> how?
17:23:34 <ayoung> or, conversley, a single API might end up calling into multiple policy rules
17:23:39 <ayoung> both are possible with the existing code
17:23:49 <rj2083> that happens already
17:23:49 <ayoung> and short of a complete audit, we won't know which...
17:23:52 <ayoung> right
17:23:58 <rj2083> and in actuality how does your spec change that?
17:24:02 <ayoung> rj2083, yes
17:24:24 <ayoung> rj2083, in my spec, the RBAC check is based on the URL, and executed prior to policy.json enforcement
17:24:36 <rj2083> i dont see anything in your spec regarding conslidation of rbac policies accross services
17:25:02 <rj2083> then your spec would require one service to talk to each other
17:25:02 <andreaf> ayoung: so the main thing for me - from a QA PoV - is that we do not break existing users on APis
17:25:07 <ayoung> rj2083, across services?  The defaults are pretty much per service.
17:25:11 <ayoung> andreaf, agreed
17:25:22 <rj2083> you were talking just now about services calling each other
17:25:33 <ayoung> andreaf, the way my spec is slated to be implemented will have a no-effect until changes are actually made by the end user
17:25:49 <rj2083> that's too broad of a statement to base anything off of
17:25:50 <ayoung> rj2083, not even there,  you can have multiple policy enforcement points in the same API
17:26:05 <rj2083> yes you can, but you should still have tests to ensure they're enforced
17:26:13 <rj2083> what youre saying right now changes nothing in patrole
17:26:18 <rj2083> or in this spec at all
17:26:46 <rj2083> if you don't want to change the default policies
17:26:53 <rj2083> no one is forcing you to run patrole tests
17:26:54 <ayoung> rj2083, if you reread the details of this spec, you see that the tests are talking about testing policy, but switching roles
17:27:05 <ayoung> that is what needs to change
17:27:18 <rj2083> switching roles is part of the setup for tests
17:27:31 <rj2083> not part of the crieteria for passing/failing the tests
17:27:54 <andreaf> ayoung, rj2083: my proposal would be to revise the spec, and include in there what the tests will gating exactly, to make sure that we don't block development of the keystone spec, as long as the "do not break users" stays
17:28:00 <ayoung> rj2083, then change the spec to be based on the URLs, and not the policy rules
17:28:22 <ayoung> @rbac_rule_validation.action(
17:28:22 <ayoung> component="Compute",
17:28:23 <ayoung> rule="compute_extension:services")
17:28:25 <ayoung> should become
17:28:25 <rj2083> that's implied n the spec
17:28:26 <andreaf> because that's not in the spec at the moment, so it's not clear what the tests would gate and how, and that's an important part of the story I think
17:28:35 <rj2083> that we wpon't be breaking user api flow
17:28:52 <ayoung> @rbac_rule_validation.action(
17:28:52 <ayoung> component="Compute",
17:28:52 <ayoung> VERB="POST"  URL="/v2/service/blah")
17:29:15 <ayoung> and that is a perfectly find way to go about doing it
17:29:30 <DavidPurcell> I think this is why Oomichi wanted Patrole as a separate plugin rather than part of Tempest.  As it stands, it can be used by operators that want it (several LCOOs included).
17:29:37 <rj2083> right
17:30:02 <ayoung> the thing that the policy approach does not provide is a way to match the URL to the policy rule.  An end user knows the URL.  Or can figure it out from the client, etc
17:30:12 <jordanP> (lcoo == Large Contributing OpenStack Operators)
17:30:18 <rj2083> the action dictates the api route
17:30:28 <DavidPurcell> jordanP: Thanks :)
17:31:04 <ayoung> The other thing to be wary of is that, with RBAC, end users should be defining a larger set of Roles
17:31:07 <andreaf> ayoung, rj2083, DavidPurcell, jordanP: so can we agree on an update on the spec?
17:31:14 <ayoung> the basic set we expect is to be:
17:31:19 <rj2083> no
17:31:23 <rj2083> spec is fine
17:31:26 <rj2083> no need to update
17:31:31 <jordanP> I am not sure what to say, but at least we should wait :)
17:31:32 <andreaf> we can clarify that such tests won't run in the integration gate
17:31:33 <ayoung> rj2083, spec needs to be updated
17:31:43 <rj2083> i see no valid use case to update it other than to state obvious things man
17:31:44 <ayoung> do not do role checks against policuy rules
17:31:51 <ayoung> you will lock us in to the existing set of roles
17:31:57 <ayoung> and that is absolutelty wrong
17:32:07 <rj2083> no one's forcing a single set or exsiting roles
17:32:09 <ayoung> There is exactly one role you can bank on
17:32:12 <ayoung> and that is admin
17:32:26 <ayoung> test RBAC against APIs, not policy rules
17:32:31 <rj2083> we're submitting changes to every project to make admin-ness properly scoped
17:32:37 <rj2083> the fact that admin is wehre it is today
17:32:44 <rj2083> is because patrole didnt exist before
17:32:56 <rj2083> and every large operator that's tried to do proper rbac
17:33:01 <rj2083> has given up because of exactly this
17:33:10 <rj2083> not because the policies weren't centralized
17:33:26 <rj2083> but because changing them has no effect because admin is hardcoded into the code in some cases
17:33:33 <ayoung> rj2083, heh
17:33:41 <ayoung> you and I are tilting at the same windmills
17:33:46 <ayoung> they are definietly GIANTS
17:34:03 <rj2083> right
17:34:07 <rj2083> i agree
17:34:16 <rj2083> i just dont think the specs have anything to do with each other
17:34:23 <rj2083> and i've seen nothing here that chagnes that
17:34:23 <andreaf> rj2083, ayoung: ok folks, I thing everyone had space to make their point, we should continue the discussion on an IRC channel or on the spec - or if needed take it to the mailing list
17:34:36 <ayoung> OK...I'm gone for then next 2 weeks
17:34:47 <rj2083> i'll be on mailing list :)
17:34:52 <rj2083> hit me up lets talk
17:34:53 <rj2083> :)
17:35:00 <andreaf> :)
17:35:14 <DavidPurcell> same for this week, then I will likely be gone for a week.
17:35:19 <andreaf> we need to continue with the QA meeting with the 25min left, sorry to cut the discussion
17:35:36 <andreaf> I think a lot of folks will be away for some time now
17:36:29 <andreaf> #action rj2083 to write to the mailing list about RBAC testing
17:36:58 <andreaf> ok anything else on specs?
17:37:38 <jordanP> that would be great, because I have a hard time understanding what you're proposing, altough it's clear you don't agree with each other
17:37:57 <andreaf> From my side I started again looking at #link https://review.openstack.org/#/c/349730 - tempest stable fixtures
17:39:09 <jordanP> about it, there might be an alternative
17:39:32 <jordanP> consistently use helpers method and the addCleanup mecanisms
17:39:32 <andreaf> I'm writing a PoC using testresources as suggested by lifeless - I see some value in that but also some issues - in the next year I will ask folks for reviews so we can decide if to use testresources or class level fixtures (as originally in the spec) - but for now I don't have enough to ask for reviews
17:40:18 <andreaf> jordanP: well what I was proposing there is to introduce an addCleanup that would work at class level
17:41:14 <jordanP> ok, I guess having network and credentials fixtures at class level is the best option. But we shouldn"t write let's say a "server fixture"
17:41:20 <andreaf> jordanP: but what lifeless tells me is that class level fixtures are evil, and we should just not use them, and instead use some mechanism like testresources to coordinate which resource needs to be refreshed / regenerated when
17:41:55 <andreaf> jordanP: yeah
17:41:56 <andreaf> jordanP: it
17:42:05 <andreaf> it's mainly about creds and networks and perhaps keypairs and s-g
17:42:38 <andreaf> the other thing is that plugins really need that stuff in lib, so we need to change that code anyways to make it a valid stable interface
17:42:59 <andreaf> I would like to decide either way and start the implementation before PTG
17:43:07 <jordanP> ok, I would like the resulting choice to be simple
17:43:19 <andreaf> heh right
17:43:27 <andreaf> ok I think we need to move on
17:43:55 <andreaf> #topic Tempest
17:43:58 <andreaf> do we have an update on the bug triage?
17:44:01 <rodrigods> yeah
17:44:11 <rodrigods> me and dmellado were taking a look on it
17:44:25 <rodrigods> i've filled some stuff at https://etherpad.openstack.org/p/tempest-weekly-bug-report
17:44:53 <rodrigods> since i'm kind of a noob in tempest, my approach was to take a look at old in progress bugs
17:44:59 <rodrigods> and see what is clearly outdated
17:45:02 <rodrigods> or invalid
17:45:17 <rodrigods> there were a bunch of them where the target didn't even exist anymore
17:45:55 <andreaf> nice
17:46:18 <andreaf> I see the overall number is reduced very good #link https://etherpad.openstack.org/p/tempest-weekly-bug-report
17:46:24 <andreaf> thanks for that
17:46:58 <andreaf> rodrigods: did you have a chance to update the graph? #link https://github.com/oomichi/bug-counter#current-graph
17:47:06 <rodrigods> andreaf, not yet
17:48:09 <andreaf> thanks rodrigods, dmellado for running the bug triage
17:48:24 <andreaf> anything else on Tempest?
17:48:57 <jordanP> I am working on Py3 compatibility
17:49:01 <jordanP> we are almost there :)
17:49:10 <andreaf> nice job! thank you
17:49:11 <rodrigods> andreaf, np! :)
17:49:17 <rodrigods> jordanP, ++
17:49:19 <jordanP> and we are almost finish with moving clients to lib/
17:49:20 <andreaf> do you have a topic for that?
17:49:56 <jordanP> no dedicated topic. A patch here: https://review.openstack.org/#/c/413739/
17:50:06 <andreaf> #link https://review.openstack.org/#/c/413739/
17:51:19 <jordanP> that's all for tempest I think :)
17:51:22 <andreaf> One thing I wanted to mention is that I think we should avoid churn in code code, especially in the areas of network scenario tests and class hierachy - we want to tackle those issue directlty
17:51:55 <andreaf> so I'm going to be -1 on most patches doing tiny little changes here and there unless they fix a bug or go in that direction
17:52:22 <jordanP> that seems wise indeed
17:53:02 <andreaf> ok moving on...
17:53:05 <andreaf> #topic DevStack + Grenade
17:53:29 <andreaf> #undo
17:53:30 <openstack> Removing item from minutes: #topic DevStack + Grenade
17:53:37 <andreaf> oh one last thing on tempest
17:53:37 <jordanP> I forgot to ask about the -ssh job...
17:53:55 <andreaf> yeah
17:53:56 <andreaf> so I added TLS on the -ssh job
17:54:04 <jordanP> patch was merged ?
17:54:12 <andreaf> the patch is merged now, so I think at this point we can let the periodic job check this over christmas
17:54:50 <andreaf> and in 2017 we can either make the job voting or add validation to the existing gate job dsvm-neutron-full
17:54:52 <jordanP> then ? make the -ssh job voting  or turns validation ON on the other jobs ?
17:55:06 <jordanP> second option would be best imo
17:55:50 <jordanP> problem is the -ssh job is used by many projects so we can't really remove it
17:55:56 <andreaf> jordanP: yes but it will impact pretty much everyone - so maybe we can go through having it voting for a few weeks and them change the main job
17:56:16 <andreaf> jordanP: it's only used by tempest, devstack and nova/neutron on experimental
17:56:34 <andreaf> jordanP: we can leave the job there just take it out of check for tempest
17:56:56 <jordanP> ok
17:56:58 <andreaf> jordanP: but next year anyways
17:57:01 <jordanP> yep
17:57:01 <andreaf> #topic DevStack + Grenade
17:57:08 <mmedvede> hi, I maintain IBM PowerKVM CI. Are you ok if we enable the CI voting on devstack patches?
17:57:14 <rodrigods> i have a review related to the PCI DSS tests
17:57:28 <rodrigods> has already one +2 https://review.openstack.org/#/c/377004/
17:57:56 <andreaf> #topic open discussion
17:58:19 <mmedvede> so related to my question above: It has been stable and mostly agrees with jenkins http://i.imgur.com/Vi4IsqR.png
17:58:27 <andreaf> mmedvede: sorry I can't answer that now
17:58:32 <mmedvede> maybe not a good time to ask it (many are on vacation)
17:58:42 <jordanP> yes, you may want to ask in January
17:58:47 <mmedvede> ok
17:58:57 <andreaf> #action andreaf oomichi check IBM PowerKVM CI for devstack
17:59:11 <andreaf> but yeah I would say next year would be better
17:59:33 <andreaf> rodrigods: sure #link https://review.openstack.org/#/c/377004/ needs reviews
17:59:43 <rodrigods> thanks andreaf
18:00:04 <andreaf> ok we're almost at time - anything else anyone?
18:00:38 <andreaf> thanks for joining today, happy holiday season everyone
18:00:41 <andreaf> #endmeeting