17:00:58 <mtreinish> #startmeeting qa 17:00:59 <openstack> Meeting started Thu Apr 24 17:00:58 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:04 <openstack> The meeting name has been set to 'qa' 17:01:12 <mtreinish> hi who do we have here today? 17:01:20 <marun> hi 17:01:29 <mlavalle> hi 17:01:32 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_April_24_2014_.281700_UTC.29 17:01:33 <andreaf> hi 17:01:36 <mtreinish> ^^^ Today's agenda 17:01:46 <dkranz> hi 17:02:38 <sdague> o/ 17:02:52 <mtreinish> mkoderer, afazekas: you guys around? 17:03:02 <afazekas> hi 17:03:09 <mtreinish> well let's get started 17:03:19 <mtreinish> #topic Oslo Liaison (mtreinish) 17:03:31 <mtreinish> so I brought this up during last week's meeting 17:04:01 <mtreinish> but I thought that'd I ask during the alternate time too 17:04:03 <mtreinish> #link https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons 17:04:26 <mtreinish> the oslo team is looking for a person to be a focal point on keeping up with oslo changes in the projects 17:04:35 <mtreinish> I think it would be good to have someone do that for tempest 17:04:43 <mtreinish> does anyone want to volunteer? 17:05:28 <mtreinish> ok well I'll bug people in person at summit about it then :) 17:05:38 <mtreinish> let's move onto the next topic 17:05:47 <mtreinish> #topic Summit sessions (mtreinish) 17:05:59 <mtreinish> #link https://etherpad.openstack.org/p/Juno-QA-design-summit-topics 17:06:25 <mtreinish> so we currently have only one extra session proposal 17:06:32 <mtreinish> so we need to figure out which one we can drop 17:07:18 <mtreinish> I also rejected boris-42's rally w/ tempest proposal because it wasn't in the qa program. But, thinking about it might be a good idea to have that session 17:07:21 <dkranz> mtreinish: Are "How to improve the UX of our testing tools:" and "Tempest GUI / Should we have a tempest GUI (in Horizon)?" similar? 17:07:31 <dkranz> mtreinish: I think we should have the rally session 17:07:48 <marun> ux -> user experience 17:07:52 <marun> not necessarily ui 17:08:04 <mtreinish> dkranz: I can combine them, but UX one was strictly about test tooling 17:08:08 <mtreinish> like the testr headaches 17:08:13 <dkranz> mtreinish: ok 17:08:17 <sdague> honestly, I'd rather give ux a whole block 17:08:28 <sdague> I think it would be good to talk through, and hopefully get some voluteers 17:08:42 <sdague> especially as I think testr will be on github around then 17:08:42 <mtreinish> the gui one is about making a tempest server which stores results and actually runs tempest with a client and a horizon plugin 17:08:48 <dkranz> arguably rally should be part of the qa program 17:08:53 <mtreinish> masayuki had a cool little diagram 17:08:54 <dkranz> mtreinish: ok 17:09:17 <sdague> yeh, having a rally session would be interesting to talk that through and figure out the challenges in having it more part of the fold 17:09:17 <marun> I'm wondering if the UX issue shouldn't be cross-project 17:09:30 <sdague> marun: possibly, though that ship has sailed 17:09:36 <marun> all projects suffer the same issues - the test tooling being suited more to running tests in ci than developing with them 17:09:43 <sdague> the cross project sessions were set over a week ago 17:09:48 <marun> ah, fair enough 17:09:54 <mtreinish> marun: yeah it's too late for the cross project 17:09:55 <marun> in that case, I hope I get to participate 17:10:00 <mtreinish> and I'm fine with giving it a qa slot 17:10:22 <andreaf> do we have anything about tempest outside of devstack / tempest for multinode which is functional to --> tempest in toci? 17:10:23 <marun> and +1 for scheduling it 17:10:38 <mtreinish> marun: well I can move it around a bit so it won't overlap 17:10:47 <mtreinish> for you hopefully 17:10:47 <marun> mtreinish: awesome 17:11:00 <dkranz> mtreinish: Maybe Marc's two could be combined? 17:11:01 <mtreinish> andreaf: the closest one on that topic would be the gui one 17:11:06 <mtreinish> dkranz: yeah I was thinking that 17:11:08 <mtreinish> they are related 17:11:25 <mtreinish> andreaf: masayuki's idea is about making tempest easy for operators 17:11:26 <dkranz> mtreinish: One is kind of a sub-case of the other. 17:11:53 <mtreinish> dkranz: yeah that's a better description because fuzz testing will use the negative framework 17:11:55 <sdague> mtreinish: so if we combine marc's 2 17:12:05 <mtreinish> then it's drop 1 17:12:12 <sdague> and we drop service debug, and bring in rally, does that make us good? 17:12:15 <mtreinish> and I'll give up my service debug api one 17:12:21 <dkranz> +1 17:12:26 <mtreinish> yeah we should be all set then 17:12:31 <sdague> cool 17:13:00 <mtreinish> ok, then the session list has been finalized, unless anyone has something else to add on it 17:13:09 <dkranz> sdague: BTW, with regard to your branchless tempest item, has that idea been broadly accepted at this point? 17:13:22 <mtreinish> I'll schedule things after the meeting 17:13:23 <sdague> dkranz: well, it's in effect 17:13:27 <dkranz> sdague: :) 17:13:31 <sdague> I mostly got head nodding 17:13:37 <dkranz> sdague: cool 17:13:52 <marun> i would nod, but i don't actually understand the implications well enough 17:13:53 <sdague> we'll find out the first time when someone tries to land juno specific test 17:14:01 <marun> i'm looking forward to hearing more at summit 17:14:47 <sdague> marun: the qa-spec hopefully covers most of the major bits https://github.com/openstack/qa-specs/blob/master/specs/branchless-tempest.rst 17:14:53 <sdague> but we'll definitely talk at summit 17:15:37 <mtreinish> ok, let's move onto the next topic 17:15:43 <marun> sdague: cool 17:15:49 <mtreinish> #topic Specs Review 17:16:10 <mtreinish> ok so we don't have any people who put there specs on the agenda for this week 17:16:26 <mtreinish> we've got a handful of open specs for review 17:16:32 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z 17:16:56 <mtreinish> so if people could take a look when they get a chance that would be cool 17:17:10 <andreaf> mtreinish: I fixed syntax issues in my qa-specs, but I still have to address the comments 17:17:10 <mtreinish> does anyone have a spec review that they would like to discuss? 17:17:15 <dkranz> mtreinish: I have a question about my non-admin spec. 17:17:25 <mtreinish> andreaf: ok 17:17:31 <mtreinish> dkranz: ok, go ahead 17:17:45 <dkranz> mtreinish: There was a comment from andreaf that we should have a network admin config option. 17:18:10 <dkranz> mtreinish: And you and I had discussed removing the one for compute because it was not used. 17:18:26 <mtreinish> a network admin config option? what does it specify? 17:18:27 <dkranz> mtreinish: At some point hopefully tempest could have its own domain as an option for admin tests (some) 17:18:37 <sdague> dkranz: so there are 2 classes of admin that we need to worry about right? 17:18:43 <sdague> there is *all* admin 17:18:46 <dkranz> sdague: At least 2 17:18:48 <sdague> and keystone admin 17:19:07 <dkranz> sdague: Yes, and this is complicated because services don't really deal with this 17:19:07 <sdague> and in keystone v3 we could be admin of a domain 17:19:15 <andreaf> dkranz, mtreinish, sdague: for identity service there is an overall admin and a domain admin (with v3) 17:19:28 <dkranz> sdague: For example, should 'nova list --all-tenants' return *all* or all in the domain? 17:19:28 <sdague> andreaf: right 17:19:30 <andreaf> but it is possible to define admin for services as well 17:19:51 <dkranz> So this will get complicated and need to be hashed out. 17:20:06 <dkranz> But I don't want this to get in the way of the capability to run non-admin now. 17:20:09 <sdague> dkranz: so I feel like we need @test.admin('....') 17:20:16 <sdague> that specifies the kind 17:20:19 <sdague> of admin needed 17:20:43 <andreaf> sdague: the list of roles would be useful 17:20:55 <sdague> the point of the qa-spec is to actually hash it out in advance so the implementation can be pretty straight forward :) 17:21:00 <afazekas> The '....' should be admin capabilities ? 17:21:04 <sdague> afazekas: yeh 17:21:10 <sdague> like what it needs 17:21:22 <mtreinish> sdague: but how do we specify the valid things in the config file 17:21:25 <sdague> honestly, I haven't thought that part all the way thorugh 17:21:48 <dkranz> sdague: ok, I will think a little more about this. 17:21:57 <sdague> that's a good question, these should be clarified in the spec 17:22:01 <andreaf> roles and policies are heavily customizable and if we want tempest to be portable to real clouds we need something like list of roles a tests depends one 17:22:08 <dkranz> sdague: The trouble is that it is perfectly clear for what we have now. 17:22:34 <sdague> dkranz: not really, there are lots of different admin creds in our config 17:22:34 <dkranz> sdague: It just doesn't work 17:23:04 <dkranz> sdague: So where should we draw the line? 17:23:15 <mtreinish> andreaf: yeah but we need to have a way to tell tempest what it has so we can check the list of roles against it 17:23:22 <dkranz> sdague: Wait until everything supports keystone v3 to do anything? 17:23:40 <sdague> dkranz: we can cut at keystone admin and service admin today 17:23:58 <sdague> honestly, I feel like we can dump all the service admins into a single bucket 17:24:02 <sdague> to begin with 17:24:05 <dkranz> sdague: ok, I'm good with that. 17:24:06 <afazekas> andreaf: there are to many capabilities defined in the services, and possible capabilities (in the policy.jsons) are frequently changing 17:24:18 <dkranz> sdague: It is a very small change from what I had 17:24:31 <sdague> dkranz: but we should have sample syntax on the decorator 17:24:41 <sdague> in the spec 17:25:07 <andreaf> sdague, dkranz: we could introduce service admin roles in devstack 17:25:14 <sdague> we could 17:25:29 <sdague> honestly, lets make this spec clear enough for someone to implement 17:25:53 <sdague> because right now, it's still missing some details, as seen by this conversation 17:26:29 <andreaf> sdague: ok - that's perhaps a dofferent spec - to have something more complex that a single admin to rule them all 17:26:51 <sdague> we also have to address tenant isolation in non admin environment 17:27:11 <sdague> I feel like that's still missing, and maybe it's not actually dkranz's spec, but it needs to happen soon 17:27:26 <sdague> because we need to stop running serial, as it keeps hiding bugs 17:27:36 <sdague> even in non admin environments 17:27:41 <mtreinish> sdague: we depend on domains for that right? 17:27:49 <mtreinish> which means we have to land the v3 auth stuff first 17:27:55 <sdague> we don't need to 17:28:02 <sdague> we can do it with preallocted users 17:28:25 <sdague> which, honestly, a service provider would probably find much more palatable 17:28:42 <mtreinish> so we set the concurrency = the number of predefined tenants 17:29:27 <mtreinish> that's an interesting sync problem between putting something in the conf file and using it in the .testr.conf 17:29:35 <mtreinish> unless we leave it up to the user to do it correctly 17:30:07 <mtreinish> sdague: but yeah I feel like that's a different spec 17:30:12 <dkranz> sdague: These are all good ideas. I mentioned some of this in the spec but did not think we were going to address the whole ball of wax right now. 17:30:17 <afazekas> Maybe some should ask for good practices for v3 roles and admin users and policies... on the ML 17:30:27 <sdague> afazekas: good thought 17:30:34 <sdague> afazekas: you want to send that out? 17:31:03 <afazekas> sdague: I would recommend a better writer :) 17:31:47 <andreaf> afazekas: I can try if you want, not that my english is fantastic though :D 17:32:02 <afazekas> andreaf: thx 17:32:14 <mtreinish> #action andreaf to write a ML post asking about good v3 keystone practices 17:32:41 <dkranz> mtreinish: I thought about the multiple users but am not sure how the testr processes get mapped. 17:33:11 <mtreinish> dkranz: I actually made a mistake above, the concurrency is set in the cli for testr 17:33:20 <mtreinish> by default it equals # cpus 17:33:30 <dkranz> mtreinish: Yes but we have to map to user/tenant names somehow 17:33:49 <sdague> so I guess this actually opens up a meta question 17:33:50 <dkranz> mtreinish: which means each tempest process has to have a different config 17:33:59 <sdague> we clearly have a longer list of things that we want in tempest 17:34:01 <mtreinish> dkranz: not necessarily 17:34:19 <sdague> that none of the most active people are working through yet 17:34:26 <sdague> this as a good example 17:34:50 <sdague> what would be a good mechanism for listing / signalling these wants 17:35:31 <dkranz> sdague: I suggest allowing "specs" that are not fully formed and do not yet have an implementer 17:35:34 <andreaf> sdague: perhaps the qa-specs are 17:35:43 <andreaf> dkranz: +1 17:35:45 <mtreinish> sdague: I'm not sure we could keep an etherpad, because it's kinda nebulous 17:35:58 <sdague> mtreinish: yeh, etherpad seems not good 17:36:04 <dkranz> sdague: But we need to use a collaboration tool, not gerrit to flesh them out. Google docs for example. 17:36:14 <sdague> what about a single page list in qa-specs ? 17:36:25 <sdague> basically paragraph level description 17:36:28 <dkranz> sdague: pointing to google docs? 17:36:36 <sdague> no, not pointing to google docs :) 17:36:38 <sdague> in qa-spec 17:36:57 <mtreinish> sdague: so a spec with a list of TODO things 17:37:03 <dkranz> sdague: ok, but how do we collaboratively evolve them? gerrit sucks for this. 17:37:03 <sdague> like a TODO.rst (better name tbd) 17:37:16 <sdague> dkranz: I actually like gerrit for this 17:37:36 <sdague> what are the concerns you have with gerrit on this? 17:37:37 <mtreinish> dkranz: when someone is ready to implement it we break it off into it's own spec 17:37:41 <dkranz> sdague: It does not support multiple writers 17:37:57 <sdague> dkranz: you can just grab the review and push your own rev 17:38:32 <dkranz> sdague: I know. But we are moving to the kind of process that zillions of people use, and there are tools designed for that. 17:38:50 <dkranz> sdague: Not code review systems 17:38:59 <dkranz> sdague: I don;'t think we should argue about this now. 17:38:59 <sdague> dkranz: do you think that zillions of people with simultaneous cursors makes it more clear? 17:39:03 <sdague> :) 17:39:05 <sdague> that's fine 17:39:19 <mtreinish> sdague: why don't you take it to the ML? 17:39:19 <sdague> ok, so tabling that 17:39:22 <sdague> sure 17:39:28 <dkranz> sdague: not simultaneous, though I have done that :) 17:39:29 <mtreinish> because I'm sure other people have ideas on this 17:39:53 <sdague> yep 17:39:58 <sdague> action me up 17:40:13 <mtreinish> #action sdague to send a ML post about how to maintain a list of TODO items 17:40:24 <dkranz> sdague: sorry, I meant zillions of users across all things done, not zillions on a single document 17:40:49 <mtreinish> ok, is there anything else on specs review? 17:41:20 <mtreinish> ok then moving on 17:41:22 <mtreinish> #topic Blueprints 17:41:37 <mtreinish> julien-llp: you had a bp on the agenda that you wanted to talk about 17:41:44 <julien-llp> yes, hi everyone 17:42:24 <mtreinish> julien-llp: ok, go ahead 17:42:26 <julien-llp> atthe moment this blueprint is waiting in the code review system and I thought that it could be a good idea to discuss a bit about it's relevancy 17:42:41 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip 17:43:02 <julien-llp> originally this blueprint was meant to implement a stress test for checking the capability of a stack to launch (have ahve fully availabl for a user) a large number of servers 17:43:33 <mtreinish> julien-llp: you need to open a spec on qa-specs for the bp. We moved to a new process for bps 17:43:44 <julien-llp> for the time being (and with my comprehension of tempest growing), I believed that this test can be used not only for in house stress tests but also a a non-regression test scenario 17:43:56 <mtreinish> julien-llp: https://github.com/openstack/qa-specs/blob/master/README.rst 17:44:00 <julien-llp> oh, ok, I'm not accustomed to this process by now :) 17:44:14 <mtreinish> julien-llp: we just started it for juno 17:44:33 <julien-llp> allright, I will add it to this document so 17:44:52 <mtreinish> julien-llp: so you want to take your stress test and make it a normal scenario test 17:44:55 <mtreinish> ? 17:45:05 <julien-llp> I implemented it that way, yes 17:45:37 <mtreinish> oh I looked at the code review I understand now 17:45:47 <mtreinish> you can just make that an api test 17:45:50 <julien-llp> I believe this is relevant to test some small batches of instances since it's a fairly common use case, and once upun a time there was somes issues in Openstack about this 17:45:57 <mtreinish> and use the decorator to use it in the stress suite 17:46:05 <mtreinish> ping me after the meeting and we can discuss it on -qa 17:46:11 <julien-llp> allright 17:46:26 <mtreinish> ok does anyone else have any bp to discuss? 17:46:40 <andreaf> yes 17:46:41 <sdague> branchless tempest is going well 17:46:55 <mtreinish> andreaf: ok, go ahead 17:46:57 <sdague> we're enforcing correctly on icehouse, and seem to have all the major holes covered 17:47:05 <andreaf> multi-auth 17:47:08 <mtreinish> sdague: cool 17:47:37 <mtreinish> sdague: have we had any backports merged with the new process? 17:47:56 <sdague> not yet, realistically the code queues are slow right now 17:47:57 <andreaf> I rebased the pending changes and addressed comments, and also I fixed the unit tests forhttps://review.openstack.org/#/c/82911/ so there are about 8 patchset available now 17:48:28 <mtreinish> andreaf: ok cool, I'll try to take a look at them today 17:48:33 <andreaf> so next step I started working on is provide support for v3 in scenario tests, but I hit an issue 17:48:49 <mtreinish> v3 support in the clients right? :) 17:49:03 <andreaf> the problem is I can create most client with token and url 17:49:13 <andreaf> but what if the token expires? 17:49:38 <andreaf> we would need a way to globally handle 401 from clients, similar issue as with handling of rate limiting 17:50:13 <andreaf> the only solution I can think of is to get that implemented in the clients - so to get v3 in the clients 17:50:26 <afazekas> andreaf: the token's lifetime is 24h by default, A test case does not runs for more than 10 minute 17:50:30 <andreaf> unless someone has any good idea on this 17:50:39 <sdague> afazekas: the token lifetime is now 1h 17:50:44 <sdague> by default 17:50:50 <sdague> which is why we started tripping it 17:50:55 <mtreinish> afazekas: also we can't assume the default lifetime either 17:51:13 <andreaf> sdague, afazekas: yes I did not want to make that assumption 17:51:16 <dkranz> sdague: Why can't devstack just set it to a larger number by default? 17:51:28 <sdague> dkranz: that doesn't solve real clouds 17:51:28 <dkranz> sdague: At least in the gate. 17:51:34 <sdague> we should address this 17:51:52 <sdague> andreaf: is it possible to handle this as a decorator around the clients 17:52:09 <sdague> if that exception comes up, redo the auth, then redo the call? 17:52:27 <andreaf> well monkey patching the client 17:52:47 <dkranz> Yes, that was previously proposed for rate-limiting 17:53:01 <afazekas> Can you use a still valid token to get new one ? Can we have our manager to request new token if it would expire in 5 minute ? Do we really want to support the case when the tokens lifetime is smaller then 10 minute ? 17:53:01 <andreaf> the client would redo the auth call now, but if I only give them token and service url they can't 17:53:16 <sdague> andreaf: oh, this is because of official clients? 17:53:22 <marun> maybe the auth code should be common across libraries and reusable by tempest? 17:53:23 <mtreinish> andreaf: can you generate a token using v3 with keystone client? 17:53:29 <andreaf> they would need to go to the auth provider and ask for a new token 17:53:42 <andreaf> mtreinish: yes that's what I'm doing 17:53:42 <marun> or is there some benefit from tempest having its own implementation? 17:54:04 <mtreinish> andreaf: so why not catch the exception regen the token with keystone client and pass that to a new client 17:54:11 <sdague> marun: mostly for the same reason we have our own implementation of everything, otherwise people tend to code around server bugs 17:54:12 <mtreinish> or a new instance of the client 17:54:36 <andreaf> yes 17:54:38 <marun> sdague: uh 17:54:43 <marun> sdague: if you're testing keystone, fine 17:54:45 <dkranz> Sounds like we need a proxy for the official clients 17:54:57 <marun> sdague: but i don't see why any other service would have to use custom coding 17:55:04 <sdague> marun: which is one of the things we have to do anyway 17:55:13 <marun> sdague: *sigh* 17:55:20 <andreaf> it would be good if the client could be given a call back to an auth provider 17:55:24 <marun> sdague: I hope I live to see mechanically-generated clients. 17:55:33 <andreaf> https://review.openstack.org/90166 17:55:37 <marun> sdague: this constant make-work nonsense offends my delicate sensibilities 17:55:41 <sdague> heh 17:56:43 <mkoderer> hi folks, sry for being late :( 17:56:54 <andreaf> I just published the code I'm working on - if anyone has good ideas on how to handle this please let me know - else we shall wait for clients to support v3 or an external auth provider 17:57:20 <mtreinish> andreaf: ok I'll take a look at that, I think we can discuss this in review or on -qa 17:57:21 <sdague> andreaf: cool, is this something that will come up in a summit session? 17:58:10 <sdague> 2 minutes 17:58:11 <andreaf> sdague: it could be relevant for the matrix one 17:58:36 <andreaf> sdague: I'll check if it can fit somewhere 17:58:55 <mtreinish> ok, well with <2 min let's get a list of reviews from people and close 17:58:59 <mtreinish> #topic critical reviews 17:59:10 <mtreinish> so does anyone have any reviews that they'd like to get eyes on? 17:59:28 <mlavalle> https://review.openstack.org/#/c/60008/ 18:00:08 <mtreinish> ok, well that's time 18:00:11 <mtreinish> thanks everyone 18:00:12 <afazekas> https://review.openstack.org/#/c/90100/ some of SSH connection issue happens because the instance fails to initialize the timer (kernel boot) http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIk1QLUJJT1MgYnVnXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTgzNTk2NTYwODEsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiI 18:00:12 <afazekas> n0= 18:00:22 <mtreinish> #endmeeting