17:00:58 <mtreinish> #startmeeting qa
17:00:59 <openstack> Meeting started Thu Apr 24 17:00:58 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:04 <openstack> The meeting name has been set to 'qa'
17:01:12 <mtreinish> hi who do we have here today?
17:01:20 <marun> hi
17:01:29 <mlavalle> hi
17:01:32 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_April_24_2014_.281700_UTC.29
17:01:33 <andreaf> hi
17:01:36 <mtreinish> ^^^ Today's agenda
17:01:46 <dkranz> hi
17:02:38 <sdague> o/
17:02:52 <mtreinish> mkoderer, afazekas: you guys around?
17:03:02 <afazekas> hi
17:03:09 <mtreinish> well let's get started
17:03:19 <mtreinish> #topic Oslo Liaison (mtreinish)
17:03:31 <mtreinish> so I brought this up during last week's meeting
17:04:01 <mtreinish> but I thought that'd I ask during the alternate time too
17:04:03 <mtreinish> #link https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
17:04:26 <mtreinish> the oslo team is looking for a person to be a focal point on keeping up with oslo changes in the projects
17:04:35 <mtreinish> I think it would be good to have someone do that for tempest
17:04:43 <mtreinish> does anyone want to volunteer?
17:05:28 <mtreinish> ok well I'll bug people in person at summit about it then :)
17:05:38 <mtreinish> let's move onto the next topic
17:05:47 <mtreinish> #topic Summit sessions (mtreinish)
17:05:59 <mtreinish> #link https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
17:06:25 <mtreinish> so we currently have only one extra session proposal
17:06:32 <mtreinish> so we need to figure out which one we can drop
17:07:18 <mtreinish> I also rejected boris-42's rally w/ tempest proposal because it wasn't in the qa program. But, thinking about it might be a good idea to have that session
17:07:21 <dkranz> mtreinish: Are "How to improve the UX of our testing tools:" and "Tempest GUI / Should we have a tempest GUI (in Horizon)?" similar?
17:07:31 <dkranz> mtreinish: I think we should have the rally session
17:07:48 <marun> ux -> user experience
17:07:52 <marun> not necessarily ui
17:08:04 <mtreinish> dkranz: I can combine them, but UX one was strictly about test tooling
17:08:08 <mtreinish> like the testr headaches
17:08:13 <dkranz> mtreinish: ok
17:08:17 <sdague> honestly, I'd rather give ux a whole block
17:08:28 <sdague> I think it would be good to talk through, and hopefully get some voluteers
17:08:42 <sdague> especially as I think testr will be on github around then
17:08:42 <mtreinish> the gui one is about making a tempest server which stores results and actually runs tempest with a client and a horizon plugin
17:08:48 <dkranz> arguably rally should be part of the qa program
17:08:53 <mtreinish> masayuki had a cool little diagram
17:08:54 <dkranz> mtreinish: ok
17:09:17 <sdague> yeh, having a rally session would be interesting to talk that through and figure out the challenges in having it more part of the fold
17:09:17 <marun> I'm wondering if the UX issue shouldn't be cross-project
17:09:30 <sdague> marun: possibly, though that ship has sailed
17:09:36 <marun> all projects suffer the same issues - the test tooling being suited more to running tests in ci than developing with them
17:09:43 <sdague> the cross project sessions were set over a week ago
17:09:48 <marun> ah, fair enough
17:09:54 <mtreinish> marun: yeah it's too late for the cross project
17:09:55 <marun> in that case, I hope I get to participate
17:10:00 <mtreinish> and I'm fine with giving it a qa slot
17:10:22 <andreaf> do we have anything about tempest outside of devstack / tempest for multinode which is functional to  --> tempest in toci?
17:10:23 <marun> and +1 for scheduling it
17:10:38 <mtreinish> marun: well I can move it around a bit so it won't overlap
17:10:47 <mtreinish> for you hopefully
17:10:47 <marun> mtreinish: awesome
17:11:00 <dkranz> mtreinish: Maybe Marc's two could be combined?
17:11:01 <mtreinish> andreaf: the closest one on that topic would be the gui one
17:11:06 <mtreinish> dkranz: yeah I was thinking that
17:11:08 <mtreinish> they are related
17:11:25 <mtreinish> andreaf: masayuki's idea is about making tempest easy for operators
17:11:26 <dkranz> mtreinish: One is kind of a sub-case of the other.
17:11:53 <mtreinish> dkranz: yeah that's a better description because fuzz testing will use the negative framework
17:11:55 <sdague> mtreinish: so if we combine marc's 2
17:12:05 <mtreinish> then it's drop 1
17:12:12 <sdague> and we drop service debug, and bring in rally, does that make us good?
17:12:15 <mtreinish> and I'll give up my service debug api one
17:12:21 <dkranz> +1
17:12:26 <mtreinish> yeah we should be all set then
17:12:31 <sdague> cool
17:13:00 <mtreinish> ok, then the session list has been finalized, unless anyone has something else to add on it
17:13:09 <dkranz> sdague: BTW, with regard to your branchless tempest item, has that idea been broadly accepted at this point?
17:13:22 <mtreinish> I'll schedule things after the meeting
17:13:23 <sdague> dkranz: well, it's in effect
17:13:27 <dkranz> sdague: :)
17:13:31 <sdague> I mostly got head nodding
17:13:37 <dkranz> sdague: cool
17:13:52 <marun> i would nod, but i don't actually understand the implications well enough
17:13:53 <sdague> we'll find out the first time when someone tries to land juno specific test
17:14:01 <marun> i'm looking forward to hearing more at summit
17:14:47 <sdague> marun: the qa-spec hopefully covers most of the major bits https://github.com/openstack/qa-specs/blob/master/specs/branchless-tempest.rst
17:14:53 <sdague> but we'll definitely talk at summit
17:15:37 <mtreinish> ok, let's move onto the next topic
17:15:43 <marun> sdague: cool
17:15:49 <mtreinish> #topic Specs Review
17:16:10 <mtreinish> ok so we don't have any people who put there specs on the agenda for this week
17:16:26 <mtreinish> we've got a handful of open specs for review
17:16:32 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:16:56 <mtreinish> so if people could take a look when they get a chance that would be cool
17:17:10 <andreaf> mtreinish: I fixed syntax issues in my qa-specs, but I still have to address the comments
17:17:10 <mtreinish> does anyone have a spec review that they would like to discuss?
17:17:15 <dkranz> mtreinish: I have a question about my non-admin spec.
17:17:25 <mtreinish> andreaf: ok
17:17:31 <mtreinish> dkranz: ok, go ahead
17:17:45 <dkranz> mtreinish: There was a comment from andreaf that we should have a network admin config option.
17:18:10 <dkranz> mtreinish: And you and I had discussed removing the one for compute because it was not used.
17:18:26 <mtreinish> a network admin config option? what does it specify?
17:18:27 <dkranz> mtreinish: At some point hopefully tempest could have its own domain as an option for admin tests (some)
17:18:37 <sdague> dkranz: so there are 2 classes of admin that we need to worry about right?
17:18:43 <sdague> there is *all* admin
17:18:46 <dkranz> sdague: At least 2
17:18:48 <sdague> and keystone admin
17:19:07 <dkranz> sdague: Yes, and this is complicated because services don't really deal with this
17:19:07 <sdague> and in keystone v3 we could be admin of a domain
17:19:15 <andreaf> dkranz, mtreinish, sdague: for identity service there is an overall admin and a domain admin (with v3)
17:19:28 <dkranz> sdague: For example, should 'nova list --all-tenants' return *all* or all in the domain?
17:19:28 <sdague> andreaf: right
17:19:30 <andreaf> but it is possible to define admin for services as well
17:19:51 <dkranz> So this will get complicated and need to be hashed out.
17:20:06 <dkranz> But I don't want this to get in the way of the capability to run non-admin now.
17:20:09 <sdague> dkranz: so I feel like we need @test.admin('....')
17:20:16 <sdague> that specifies the kind
17:20:19 <sdague> of admin needed
17:20:43 <andreaf> sdague: the list of roles would be useful
17:20:55 <sdague> the point of the qa-spec is to actually hash it out in advance so the implementation can be pretty straight forward :)
17:21:00 <afazekas> The '....' should be admin capabilities ?
17:21:04 <sdague> afazekas: yeh
17:21:10 <sdague> like what it needs
17:21:22 <mtreinish> sdague: but how do we specify the valid things in the config file
17:21:25 <sdague> honestly, I haven't thought that part all the way thorugh
17:21:48 <dkranz> sdague: ok, I will think a little more about this.
17:21:57 <sdague> that's a good question, these should be clarified in the spec
17:22:01 <andreaf> roles and policies are heavily customizable and if we want tempest to be portable to real clouds we need something like list of roles a tests depends one
17:22:08 <dkranz> sdague: The trouble  is that it is perfectly clear for what we have now.
17:22:34 <sdague> dkranz: not really, there are lots of different admin creds in our config
17:22:34 <dkranz> sdague: It just doesn't work
17:23:04 <dkranz> sdague: So where should we draw the line?
17:23:15 <mtreinish> andreaf: yeah but we need to have a way to tell tempest what it has so we can check the list of roles against it
17:23:22 <dkranz> sdague: Wait until everything supports keystone v3 to do anything?
17:23:40 <sdague> dkranz: we can cut at keystone admin and service admin today
17:23:58 <sdague> honestly, I feel like we can dump all the service admins into a single bucket
17:24:02 <sdague> to begin with
17:24:05 <dkranz> sdague: ok, I'm good with that.
17:24:06 <afazekas> andreaf: there are to many capabilities defined in the services, and possible capabilities (in the policy.jsons)  are frequently changing
17:24:18 <dkranz> sdague: It is a very small change from what I had
17:24:31 <sdague> dkranz: but we should have sample syntax on the decorator
17:24:41 <sdague> in the spec
17:25:07 <andreaf> sdague, dkranz: we could introduce service admin roles in devstack
17:25:14 <sdague> we could
17:25:29 <sdague> honestly, lets make this spec clear enough for someone to implement
17:25:53 <sdague> because right now, it's still missing some details, as seen by this conversation
17:26:29 <andreaf> sdague: ok - that's perhaps a dofferent spec - to have something more complex that a single admin to rule them all
17:26:51 <sdague> we also have to address tenant isolation in non admin environment
17:27:11 <sdague> I feel like that's still missing, and maybe it's not actually dkranz's spec, but it needs to happen soon
17:27:26 <sdague> because we need to stop running serial, as it keeps hiding bugs
17:27:36 <sdague> even in non admin environments
17:27:41 <mtreinish> sdague: we depend on domains for that right?
17:27:49 <mtreinish> which means we have to land the v3 auth stuff first
17:27:55 <sdague> we don't need to
17:28:02 <sdague> we can do it with preallocted users
17:28:25 <sdague> which, honestly, a service provider would probably find much more palatable
17:28:42 <mtreinish> so we set the concurrency = the number of predefined tenants
17:29:27 <mtreinish> that's an interesting sync problem between putting something in the conf file and using it in the .testr.conf
17:29:35 <mtreinish> unless we leave it up to the user to do it correctly
17:30:07 <mtreinish> sdague: but yeah I feel like that's a different spec
17:30:12 <dkranz> sdague: These are all good ideas. I mentioned some of this in the spec but did not think we were going to address the whole ball of wax right now.
17:30:17 <afazekas> Maybe some should ask for good practices for v3  roles and admin users and policies... on the ML
17:30:27 <sdague> afazekas: good thought
17:30:34 <sdague> afazekas: you want to send that out?
17:31:03 <afazekas> sdague: I would recommend a better writer :)
17:31:47 <andreaf> afazekas: I can try if you want, not that my english is fantastic though :D
17:32:02 <afazekas> andreaf: thx
17:32:14 <mtreinish> #action andreaf to write a ML post asking about good v3 keystone practices
17:32:41 <dkranz> mtreinish: I thought about the multiple users but am not sure how the testr processes get mapped.
17:33:11 <mtreinish> dkranz: I actually made a mistake above, the concurrency is set in the cli for testr
17:33:20 <mtreinish> by default it equals # cpus
17:33:30 <dkranz> mtreinish: Yes but we have to map to user/tenant names somehow
17:33:49 <sdague> so I guess this actually opens up a meta question
17:33:50 <dkranz> mtreinish: which means each tempest process has to have a different config
17:33:59 <sdague> we clearly have a longer list of things that we want in tempest
17:34:01 <mtreinish> dkranz: not necessarily
17:34:19 <sdague> that none of the most active people are working through yet
17:34:26 <sdague> this as a good example
17:34:50 <sdague> what would be a good mechanism for listing / signalling these wants
17:35:31 <dkranz> sdague: I suggest allowing "specs" that are not fully formed and do not yet have an implementer
17:35:34 <andreaf> sdague: perhaps the qa-specs are
17:35:43 <andreaf> dkranz: +1
17:35:45 <mtreinish> sdague: I'm not sure we could keep an etherpad, because it's kinda nebulous
17:35:58 <sdague> mtreinish: yeh, etherpad seems not good
17:36:04 <dkranz> sdague: But we need to use a collaboration tool, not gerrit to flesh them out. Google docs for example.
17:36:14 <sdague> what about a single page list in qa-specs ?
17:36:25 <sdague> basically paragraph level description
17:36:28 <dkranz> sdague: pointing to google docs?
17:36:36 <sdague> no, not pointing to google docs :)
17:36:38 <sdague> in qa-spec
17:36:57 <mtreinish> sdague: so a spec with a list of TODO things
17:37:03 <dkranz> sdague: ok, but how do we collaboratively evolve them? gerrit sucks for this.
17:37:03 <sdague> like a TODO.rst (better name tbd)
17:37:16 <sdague> dkranz: I actually like gerrit for this
17:37:36 <sdague> what are the concerns you have with gerrit on this?
17:37:37 <mtreinish> dkranz: when someone is ready to implement it we break it off into it's own spec
17:37:41 <dkranz> sdague: It does not support multiple writers
17:37:57 <sdague> dkranz: you can just grab the review and push your own rev
17:38:32 <dkranz> sdague: I know. But we are moving to the kind of process that zillions of people use, and there are tools designed for that.
17:38:50 <dkranz> sdague: Not code review systems
17:38:59 <dkranz> sdague: I don;'t think we should argue about this now.
17:38:59 <sdague> dkranz: do you think that zillions of people with simultaneous cursors makes it more clear?
17:39:03 <sdague> :)
17:39:05 <sdague> that's fine
17:39:19 <mtreinish> sdague: why don't you take it to the ML?
17:39:19 <sdague> ok, so tabling that
17:39:22 <sdague> sure
17:39:28 <dkranz> sdague: not simultaneous, though I have done that :)
17:39:29 <mtreinish> because I'm sure other people have ideas on this
17:39:53 <sdague> yep
17:39:58 <sdague> action me up
17:40:13 <mtreinish> #action sdague to send a ML post about how to maintain a list of TODO items
17:40:24 <dkranz> sdague: sorry, I meant zillions of users across all things done, not zillions on a single document
17:40:49 <mtreinish> ok, is there anything else on specs review?
17:41:20 <mtreinish> ok then moving on
17:41:22 <mtreinish> #topic Blueprints
17:41:37 <mtreinish> julien-llp: you had a bp on the agenda that you wanted to talk about
17:41:44 <julien-llp> yes, hi everyone
17:42:24 <mtreinish> julien-llp: ok, go ahead
17:42:26 <julien-llp> atthe moment this blueprint is waiting in the code review system and I thought that it could be a good idea to discuss a bit about it's relevancy
17:42:41 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip
17:43:02 <julien-llp> originally this blueprint was meant to implement a stress test for checking the capability of a stack to launch (have ahve fully availabl for a user) a large number of servers
17:43:33 <mtreinish> julien-llp: you need to open a spec on qa-specs for the bp. We moved to a new process for bps
17:43:44 <julien-llp> for the time being (and with my comprehension of tempest growing), I believed that this test can be used not only for in house stress tests but also a a non-regression test scenario
17:43:56 <mtreinish> julien-llp: https://github.com/openstack/qa-specs/blob/master/README.rst
17:44:00 <julien-llp> oh, ok, I'm not accustomed to this process by now :)
17:44:14 <mtreinish> julien-llp: we just started it for juno
17:44:33 <julien-llp> allright, I will add it to this document so
17:44:52 <mtreinish> julien-llp: so you want to take your stress test and make it a normal scenario test
17:44:55 <mtreinish> ?
17:45:05 <julien-llp> I implemented it that way, yes
17:45:37 <mtreinish> oh I looked at the code review I understand now
17:45:47 <mtreinish> you can just make that an api test
17:45:50 <julien-llp> I believe this is relevant to test some small batches of instances since it's a fairly common use case, and once upun a time there was somes issues in Openstack about this
17:45:57 <mtreinish> and use the decorator to use it in the stress suite
17:46:05 <mtreinish> ping me after the meeting and we can discuss it on -qa
17:46:11 <julien-llp> allright
17:46:26 <mtreinish> ok does anyone else have any bp to discuss?
17:46:40 <andreaf> yes
17:46:41 <sdague> branchless tempest is going well
17:46:55 <mtreinish> andreaf: ok, go ahead
17:46:57 <sdague> we're enforcing correctly on icehouse, and seem to have all the major holes covered
17:47:05 <andreaf> multi-auth
17:47:08 <mtreinish> sdague: cool
17:47:37 <mtreinish> sdague: have we had any backports merged with the new process?
17:47:56 <sdague> not yet, realistically the code queues are slow right now
17:47:57 <andreaf> I rebased the pending changes and addressed comments, and also I fixed the unit tests forhttps://review.openstack.org/#/c/82911/ so  there are about 8 patchset available now
17:48:28 <mtreinish> andreaf: ok cool, I'll try to take a look at them today
17:48:33 <andreaf> so next step I started working on is provide support for v3 in scenario tests, but I hit an issue
17:48:49 <mtreinish> v3 support in the clients right? :)
17:49:03 <andreaf> the problem is I can create most client with token and url
17:49:13 <andreaf> but what if the token expires?
17:49:38 <andreaf> we would need a way to globally handle 401 from clients, similar issue as with handling of rate limiting
17:50:13 <andreaf> the only solution I can think of is to get that implemented in the clients - so to get v3 in the clients
17:50:26 <afazekas> andreaf: the token's lifetime is 24h by default, A test case does not runs for more than 10 minute
17:50:30 <andreaf> unless someone has any good idea on this
17:50:39 <sdague> afazekas: the token lifetime is now 1h
17:50:44 <sdague> by default
17:50:50 <sdague> which is why we started tripping it
17:50:55 <mtreinish> afazekas: also we can't assume the default lifetime either
17:51:13 <andreaf> sdague, afazekas: yes I did not want to make that assumption
17:51:16 <dkranz> sdague: Why can't devstack just set it to a larger number by default?
17:51:28 <sdague> dkranz: that doesn't solve real clouds
17:51:28 <dkranz> sdague: At least in the gate.
17:51:34 <sdague> we should address this
17:51:52 <sdague> andreaf: is it possible to handle this as a decorator around the clients
17:52:09 <sdague> if that exception comes up, redo the auth, then redo the call?
17:52:27 <andreaf> well monkey patching the client
17:52:47 <dkranz> Yes, that was previously proposed for rate-limiting
17:53:01 <afazekas> Can you use a still valid token to get new one ?   Can we have our manager to request new token if it would expire in 5 minute ? Do we really want to support the case when the tokens lifetime is smaller then 10 minute ?
17:53:01 <andreaf> the client would redo the auth call now, but if I only give them token and service url they can't
17:53:16 <sdague> andreaf: oh, this is because of official clients?
17:53:22 <marun> maybe the auth code should be common across libraries and reusable by tempest?
17:53:23 <mtreinish> andreaf: can you generate a token using v3 with keystone client?
17:53:29 <andreaf> they would need to go to the auth provider and ask for a new token
17:53:42 <andreaf> mtreinish: yes that's what I'm doing
17:53:42 <marun> or is there some benefit from tempest having its own implementation?
17:54:04 <mtreinish> andreaf: so why not catch the exception regen the token with keystone client and pass that to a new client
17:54:11 <sdague> marun: mostly for the same reason we have our own implementation of everything, otherwise people tend to code around server bugs
17:54:12 <mtreinish> or a new instance of the client
17:54:36 <andreaf> yes
17:54:38 <marun> sdague: uh
17:54:43 <marun> sdague: if you're testing keystone, fine
17:54:45 <dkranz> Sounds like we need a proxy for the official clients
17:54:57 <marun> sdague: but i don't see why any other service would have to use custom coding
17:55:04 <sdague> marun: which is one of the things we have to do anyway
17:55:13 <marun> sdague: *sigh*
17:55:20 <andreaf> it would be good if the client could be given a call back to an auth provider
17:55:24 <marun> sdague: I hope I live to see mechanically-generated clients.
17:55:33 <andreaf> https://review.openstack.org/90166
17:55:37 <marun> sdague: this constant make-work nonsense offends my delicate sensibilities
17:55:41 <sdague> heh
17:56:43 <mkoderer> hi folks, sry for being late :(
17:56:54 <andreaf> I just published the code I'm working on - if anyone has good ideas on how to handle this please let me know - else we shall wait for clients to support v3 or an external auth provider
17:57:20 <mtreinish> andreaf: ok I'll take a look at that, I think we can discuss this in review or on -qa
17:57:21 <sdague> andreaf: cool, is this something that will come up in a summit session?
17:58:10 <sdague> 2 minutes
17:58:11 <andreaf> sdague: it could be relevant for the matrix one
17:58:36 <andreaf> sdague: I'll check if it can fit somewhere
17:58:55 <mtreinish> ok, well with <2 min let's get a list of reviews from people and close
17:58:59 <mtreinish> #topic critical reviews
17:59:10 <mtreinish> so does anyone have any reviews that they'd like to get eyes on?
17:59:28 <mlavalle> https://review.openstack.org/#/c/60008/
18:00:08 <mtreinish> ok, well that's time
18:00:11 <mtreinish> thanks everyone
18:00:12 <afazekas> https://review.openstack.org/#/c/90100/ some of SSH connection issue happens because the instance fails to initialize the timer (kernel boot) http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIk1QLUJJT1MgYnVnXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTgzNTk2NTYwODEsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiI
18:00:12 <afazekas> n0=
18:00:22 <mtreinish> #endmeeting