22:00:14 <mtreinish> #startmeeting qa 22:00:14 <openstack> Meeting started Thu Apr 3 22:00:14 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:18 <openstack> The meeting name has been set to 'qa' 22:00:26 <mtreinish> hi who do we have here today? 22:00:28 <sdague> o/ 22:00:32 <maurosr> o/ 22:00:34 <masayukig> hi 22:00:51 <zhikunliu> o/ 22:00:52 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:00:58 <mtreinish> ^^^ Today's agenda 22:01:05 <dkranz> o/ 22:01:42 <mtreinish> ok lets get started 22:01:57 <mtreinish> #topic Summit sessions (mtreinish) 22:02:00 <cyeoh> o/ 22:02:16 <mtreinish> so I started an etherpad to track design summit proposals: 22:02:20 <mtreinish> #link https://etherpad.openstack.org/p/Juno-QA-design-summit-topics 22:02:42 <mtreinish> I thought it would be good to do what we did last cycle and have a discussion in the meeting about the sessions 22:03:02 <sdague> what else has been submitted into the tool? 22:03:03 <mtreinish> so if people had ideas for a summit session if you could put them in the etherpad 22:03:24 <mtreinish> sdague: right now there are only 2 22:03:36 <sdague> probably worth transposing them to the etherpad 22:03:45 <mtreinish> "A case for project level functional testing" and "Rally & Tempest integration (future steps)" 22:04:00 <sdague> the first one sounds like an overlap of the one dkranz put in 22:04:05 <dkranz> sdague: It is 22:04:16 <dkranz> sdague: I didn't know about the others 22:04:18 <dkranz> sdague: My bad 22:04:23 <sdague> dkranz: no worries 22:04:55 <cyeoh> I guess we should talk about how we can reduce the code overlap between v2 and v3 tests, but I doubt we need whole session on that 22:04:57 <mtreinish> dkranz: http://summit.openstack.org/cfp/details/134 22:05:25 <sdague> cyeoh: well, a refactoring for maintainability session might be good 22:05:27 <sdague> include that 22:05:30 <mtreinish> cyeoh: yeah probably not, but it could be part of a larger session 22:05:38 <sdague> plus other items people have 22:05:58 <sdague> I still kind of want to do the nice skip decorators, but that's back burner 22:06:21 <mtreinish> so anyway if people have ideas if they can put them on the etherpad and we'll go through it in a couple of weeks during the meeting 22:06:30 <sdague> yeh, probably plan for 3 weeks out 22:06:38 <sdague> as decision time 22:06:43 * mtreinish needs to look at a calendar 22:06:58 <mtreinish> ok that's all I had to talk about on this for now 22:07:01 <mtreinish> so let's move on 22:07:10 <mtreinish> #topic Blueprints 22:07:21 <mtreinish> so sdague why don't you start of on the specs stuff 22:07:25 <sdague> sure 22:07:28 <sdague> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z 22:07:37 <sdague> there are the open specs we currently have 22:08:22 <sdague> currently all of qa-core has +2 rights, which I think is right 22:08:32 <sdague> though we should probably by convention let the PTL do +A 22:08:42 <sdague> just to make sure they are in on the vote 22:08:45 <sdague> or ack it 22:09:10 <sdague> I think right now we have one that's fully ready 22:09:16 <sdague> https://review.openstack.org/#/c/83264/ 22:09:29 <sdague> and I went through the rest of them and provided some feedback on additions 22:09:34 <cyeoh> +1 for the +A policy 22:09:45 <sdague> anyone have objectsions to https://review.openstack.org/#/c/83264/ ? 22:09:58 <mtreinish> sdague: I hope not it's mostly implemented :) 22:10:05 <sdague> otherwise I'll land that one as our first one 22:10:17 <sdague> ok, done 22:10:18 <dkranz> sdague: No, but should this include an 'admin' tag or is that a different bp? 22:10:30 <mtreinish> dkranz: this is just the service tagging 22:10:39 <dkranz> mtreinish: ok 22:10:39 <mtreinish> like compute, image, etc... 22:10:50 <sdague> dkranz: I think admin clean is a different qa-spec 22:10:58 <sdague> can you do a draft on that? 22:11:00 <mtreinish> which all thats left on is going through and finding where we need to tag things 22:11:09 <sdague> I know you were very interested in that yesterday 22:11:39 <dkranz> sdague: Yes 22:12:03 <cyeoh> re: 83264 can you do a hacking check to see if someone does add a service name which does exist in the path 22:12:21 <sdague> #action dkranz to draft no admin qa-spec 22:12:23 <cyeoh> and flag an error if it occurs? Just to save inconsistencies creeping in accidentally 22:12:25 <mtreinish> cyeoh: was that not in there. I intend to it came up during the spec review 22:12:39 <cyeoh> mtreinish: cool! 22:12:41 <mtreinish> or I thought it did 22:12:59 <sdague> well, if not, we can always do an ammendment :) 22:13:14 <mtreinish> no it's in there L52 22:13:30 <mtreinish> it's just not a work item... 22:13:36 <sdague> cyeoh, dkranz: would you guys mind doing a run through the existing specs and providing comments as well 22:13:49 <sdague> I'd like to get us used to this model of making sure we have bases covered 22:13:55 <dkranz> sdague: Will do 22:14:00 <cyeoh> sdague: ok 22:14:01 <sdague> fortunately we've only got a few, so it's pretty quick 22:14:35 <mtreinish> I'm also going get the infra side setup soon 22:14:37 <sdague> #action sdague to reset all non high blueprints, point people at qa-specs 22:14:45 <mtreinish> so we'll publish the specs somewhere 22:14:49 <mtreinish> when they get merged 22:14:56 <sdague> mtreinish: I'm not sure why 22:15:00 <mtreinish> it looks pretty 22:15:04 <sdague> I really feel like the specs are ok in git 22:15:20 <sdague> because the blueprint is really the entry point 22:15:28 <cyeoh> having read through a few long nova ones, I think its easier to read in processed html 22:15:35 <sdague> cyeoh: ok 22:15:50 <sdague> then I'll defer 22:16:19 <mtreinish> sdague: ok is there anything else to discuss on the specs? 22:16:27 <sdague> only one last thing 22:16:50 <sdague> I feel like we should have ~4 specs landed as examples before we open this up to the wide world via ML 22:17:09 <sdague> so will hold off a few 22:17:24 <sdague> ok, done, we can move on 22:17:28 <mtreinish> that seems fair 22:17:47 <mtreinish> ok then let's discuss bps in progress 22:17:55 <mtreinish> zhikunliu: you put something on the agenda 22:18:00 <zhikunliu> yes 22:18:03 <zhikunliu> https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests 22:18:04 <mtreinish> go ahead 22:18:24 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests 22:18:56 <zhikunliu> we have already create basic cinder v2 tests. Not sure if we should continue to port v1 tests to v2 like nova v2 to v3 22:19:19 <mtreinish> zhikunliu: how much duplication is there between v1 and v2? 22:19:26 <mtreinish> we want to have good coverage for both 22:20:15 <zhikunliu> about 10 test files 22:20:22 <mtreinish> cyeoh and ken1ohmichi have been more plugged in with the nova v2 v3 stuff so they may have some advice 22:20:52 <ken1ohmichi> OK, I will see cinder API tests. 22:21:24 <zhikunliu> thank you! 22:21:39 <ken1ohmichi> np:) 22:21:46 <mtreinish> zhikunliu: so we want to exercise both the v1 and v2 apis. It's just a question of we go about implementing that. Inheritance vs 2 copies for example 22:21:52 <cyeoh> yea I guess it comes down to how similar the APIs look - we tried along the way but weren't that succesfull, but ken1ohmichi has some examples of a direction we'll probably take 22:22:13 <sdague> yeh, the nova v2 api is exceptionally odd though 22:22:20 <sdague> maybe we can do better here 22:22:32 <ken1ohmichi> yes, the sample is https://review.openstack.org/#/c/78137/ 22:22:55 <sdague> so I would honestly really encourage this to take place in a qa-spec - https://github.com/openstack/qa-specs because I think that provides a really good way of reviewing the approach 22:23:07 <sdague> and then we can all be agreed on approach and just review for correctness 22:23:26 <ken1ohmichi> I got it. I will propose it to qa-specs 22:23:40 <ken1ohmichi> as the one of the samples:) 22:23:43 <cyeoh> +1, though I don't think it hurts to have a bit of POC code to point to 22:23:53 <sdague> ken1ohmichi: great 22:23:56 <sdague> cyeoh: agreed 22:24:15 <mtreinish> zhikunliu: ok is there anything else on that bp? 22:24:24 <zhikunliu> no, thanks... 22:24:35 <mtreinish> ok does anyone have any other bps they'd like to bring up? 22:25:14 <sdague> if not, I'd actually like to jump to the tempest branch discussion 22:25:32 <sdague> because I want to make sure we give that enough time, for folks that we're in -qa channel yesterday to see it 22:25:36 <dkranz> sdague: I think I got through classifying about half of the failures 22:25:40 <mtreinish> ok then let's move onto that 22:25:44 <mtreinish> #topic Tempest branching strategy 22:26:14 <mtreinish> sdague: you've got the floor 22:26:30 <sdague> so background 22:26:33 <sdague> #link http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2014-04-02.log 22:26:42 <sdague> starting at about 2014-04-02T13:29:08 22:27:04 <sdague> a question came up about the state of tempest as a validation tool for the openstack brand 22:27:34 <sdague> and out of that conversation I asked the question: does tempest really need branches? 22:28:00 <sdague> because it *should* be testing an API that is invariant between releases 22:28:12 <dkranz> sdague: I think it needs versions, not branches, like client libraries. 22:28:13 <sdague> or changed in ways that are detectable/flaggable 22:28:24 <sdague> dkranz: sure, lets get to that in a sec 22:28:56 <sdague> we have 2 weeks before we would normally set the stable/icehouse branch 22:29:04 <sdague> so now is the time to decide if maybe we don't 22:29:25 <sdague> and use master tempest on stable/icehouse and master going forward, and try to get tempest master to work against stable/havana 22:29:37 <cyeoh> so I'm not saying I'm opposed to it, but my concern is if we branch of the overhead of adding new tests 22:30:06 <cyeoh> having to add them to multiple branches if we want to use them as a validation tool 22:30:08 <sdague> cyeoh: vs. the cost of backporting them? 22:30:27 <sdague> because thinks like defcore/refstack are going to lag 22:30:38 <sdague> at icehouse summit they are going to put out a havana based tool 22:30:43 <sdague> which vendors will want to fix 22:31:14 <cyeoh> they are going to want to fix the tool? 22:31:28 <dkranz> cyeoh: The tool is going to run tempest 22:31:31 <sdague> I am sure they will object to some results 22:31:43 <cyeoh> ok 22:31:53 <sdague> and they may be correct in tempest fixes that should land, and then backport 22:32:36 <sdague> the reason we set a branch today is because until this release we didn't have a feature enable mechanism 22:32:47 <sdague> so branch was a giant feature flag 22:32:54 <mtreinish> sdague: well we did but it was a mess 22:32:57 <sdague> havana nova supports X 22:33:06 <sdague> icehouse nova support X + Y 22:33:29 <sdague> however the feature flag model should mean that new extensions just come in under a feature flag, for instance 22:33:47 <mtreinish> sdague: so the prereq for this on devstack as I said yesterday 22:33:52 <dkranz> sdague: The big advantage of running master is that adding tests always lags the release 22:33:52 <sdague> correct 22:34:01 <mtreinish> we have to make sure we lock the feature set in the devstack config for stable/icehouse 22:34:15 <dkranz> sdague: RIght not there are many tests in master that can and should run when testing havana 22:34:39 <dkranz> sdague: And many that don't due to questionable api changes 22:34:46 <sdague> dkranz: sure 22:34:48 <cyeoh> so just to clarify for me how this would work - if we add Nova API V2 tests during Juno, what would we have to do to ensure they are added for both Juno Nova API testing as well as say Icehouse 22:35:17 <sdague> tempest master would gate on master and stable/icehouse of all the server projects 22:35:17 <mtreinish> cyeoh: but they should given the stable api :) 22:35:21 <dkranz> cyeoh: I would say they should run on both by default 22:35:38 <cyeoh> dkranz: yep that's what I'd like the default to be 22:35:45 <cyeoh> mtreinish: yea, they *should* :-) 22:35:45 <sdague> I've been working on devstack gate branch overrides that would let us specify that behavior 22:35:52 <dkranz> cyeoh: And if a test doesn't run due to a havana bug that was fixed iceshouse we tag it as >havana 22:36:00 <sdague> cyeoh: so it will only be 1 add to tempest 22:36:06 <sdague> and it will gate on both branches 22:36:12 <sdague> so you know if it works or not 22:36:22 <cyeoh> sdague: ok, sounds great then :-) 22:36:30 <dkranz> sdague: So are we going to double the gate jobs ? 22:36:37 <mtreinish> dkranz: we have to 22:36:38 <sdague> for tempest 22:36:48 <dkranz> sdague: Actually triple since havana + icehouse + master 22:36:55 <dkranz> sdague: That is my biggest concern with this approach 22:36:58 <mtreinish> dkranz: this will only be for icehouse forward 22:37:02 <sdague> it's only on tempest 22:37:12 <dkranz> sdague: Ah, ok 22:37:19 <sdague> mtreinish: I think getting havana over is still on the table 22:37:23 <sdague> depending on how much work it is 22:37:32 <mtreinish> it is but the discussion now is about what we do on the 17th 22:37:41 <mtreinish> we still have the havana branch whether we like it or not 22:37:46 <sdague> right, for the 17th, it would be just icehouse + master 22:37:46 <dkranz> mtreinish: I don't see why we have to commit 22:37:51 <sdague> agreed 22:38:17 <sdague> I'll put the havana jobs on nv, and if we get it working, we'll delete the havana branch, and win 22:38:23 <sdague> if not 22:38:28 <sdague> it ages out in 5 months 22:38:54 <dkranz> sdague: But it can't if refstack is using it 22:39:04 <dkranz> sdague: I mean can't age out 22:39:09 <sdague> havana still stops being supported in 5 months 22:39:15 <mtreinish> dkranz: havana goes eol in 5 months 22:39:17 <sdague> openstack policy hasn't changed in that regard 22:39:21 <mtreinish> we just did it for grizzly 22:39:48 <dkranz> sdague: That doesn't mean refstack will stop using it. Have you told them this intent? 22:39:57 <dkranz> sdague: Cause it makes no sense from their standpoint 22:40:09 <dkranz> sdague: Of course it could just be cloned into a refstack repo at that point. 22:40:26 <sdague> dkranz: honestly, I'm not policing them, I have lots of other things to do :) 22:40:36 <dkranz> sdague: RIght :) 22:40:53 <dkranz> sdague: Based on my results so far I think it is very doable to get master to work with havana 22:40:58 <sdague> ken1ohmichi, masayukig any thoughts on this one? I'd like to know if there are other problems we haven't thought about yet 22:41:02 <sdague> dkranz: great 22:41:14 <dkranz> sdague: We just need a tagging strategy for the breaks due to bug fixes in icehouse and api changes 22:41:16 <sdague> I should have gate jobs running nv tomorrow that will let us do that 22:41:39 <sdague> dkranz: example? 22:41:54 <dkranz> sdague: https://etherpad.openstack.org/p/master-against-havana-errors 22:42:26 <ken1ohmichi> honestly, I am not familar with refstack. 22:42:33 <sdague> dkranz: I was wondering about "tagging strategy" 22:42:51 <sdague> ken1ohmichi: refstack is basically a website that some foundation folks will run where you can submit tempest tests 22:42:52 <dkranz> ken1ohmichi: It is the "use tempest to gate use of OpenStack trademark" 22:43:01 <sdague> test results 22:43:04 <sdague> for your product 22:43:16 <sdague> as a piece of trademark use policy 22:43:28 <sdague> they are still working out all the trademark details 22:43:35 <dkranz> sdague: about tagging strategy, I think we should have a tag saying a test is enabled as of some version 22:43:49 <sdague> running a subset of tempest successfully will be a part of what you need to do to get the trademark 22:43:50 <mtreinish> dkranz: that should be handled by config 22:44:00 <sdague> right, I agree with mtreinish 22:44:05 <ken1ohmichi> oh, thanks. I got it little. 22:44:07 <maurosr> sdague: one thing that came to my mind is the maintenance burden, like in this situation we cant remove nova's xml tests 22:44:20 <dkranz> mtreinish: how? some of the tests have different havana/icehouse behavior due to api change 22:44:20 <maurosr> (not sure if you guys touched this matter, lost my connection) 22:44:27 <masayukig> sdague: Does refstack still uses grizzly? 22:44:28 <sdague> maurosr: that's true, at least not for a while 22:44:38 <sdague> masayukig: they are building on havana atm 22:44:45 <sdague> from what I understand 22:45:04 <mtreinish> dkranz: well those types of changes will be blocked by tempest in the first place 22:45:32 <mtreinish> it really only becomes a problem for when we add new test coverage 22:45:45 <dkranz> mtreinish: which we did in icehouse 22:45:47 <maurosr> since api changes are not supposed to happen I guess that is a 'good break' 22:45:59 <dkranz> mtreinish: And the tests were not blocked. We changed them. 22:46:08 <dkranz> mtreinish: YOu can see examples in that etherpad 22:46:10 <maurosr> it's something that maybe cant work planty right now, but will be good at long term 22:46:11 <mtreinish> dkranz: yes because we default everything on everywhere 22:46:24 <mtreinish> but if we have a static set of what's expected in devstack for icehouse 22:46:30 <mtreinish> then those new tests won't run 22:46:41 <sdague> dkranz: there are definitely details here on what the 2 step is, but I think those are doable 22:46:50 <dkranz> mtreinish: But why shouldn't a new test for a stable API in a previous version run? 22:47:04 <dkranz> against the old version 22:47:14 <dkranz> mtreinish: I agree we can do better icehouse->juno 22:47:26 <mtreinish> dkranz: it should but we'd need a devstack change to add it to the stable job (it's just a shift in the 2 step) 22:47:32 <sdague> so I think what we probably want to do is after stable/icehouse on devstack is set 22:47:52 <sdague> we generate the full extension list and hardcode it in devstack stable/icehouse 22:47:55 <sdague> but it floats on master 22:48:01 <sdague> so stuff is auto added 22:48:07 <mtreinish> sdague: exactly 22:48:19 <sdague> but that's actually after the 17th 22:48:36 <mtreinish> yeah it would have to be because we need the stable branch on devstack 22:48:38 <dkranz> sdague: That's fine but half the problem. The other part is stuff that actually changed, or that works in icehouse but not havana due to an unfixed bug in havana 22:48:48 <sdague> dkranz: sure 22:48:56 <dkranz> sdague: At a minimum we could just skip all the fails I found for havana. 22:49:08 <dkranz> sdague: That would be a better test than what is on stable/havana now. 22:49:19 <sdague> well, I don't want to branch skip 22:49:24 <sdague> I think that logic gets nutty 22:49:36 <sdague> so we either fix it, feature flag it, or the job doesn't vote 22:49:58 <dkranz> sdague: Why? We just classify the test as belonging to a new feature. Just a funny one. 22:50:18 <sdague> because we're already going to have 200 features 22:50:23 <sdague> that are real ones 22:50:28 <dkranz> sdague: And this would be one more. 22:50:46 <sdague> I think if we invent funny ones that we can't double check on real extension lists we're going to get it wrong 22:50:57 <sdague> there is a big who watches the watcher thing here 22:51:02 <dkranz> sdague: Anyway, the goal is to run master against havana with as many tests as we can get to work. 22:51:10 <dkranz> sdague: We need to figure the details but not right now. 22:51:14 <sdague> sure 22:51:21 <ken1ohmichi> sdague: the extension list means API list? 22:51:29 <sdague> ken1ohmichi: yes 22:51:43 <mtreinish> ken1ohmichi: yeah it's the api_extensions config option 22:51:57 <dkranz> sdague: What I am talking about is more like bug skipping than feature config. 22:52:05 <ken1ohmichi> sdague: ok, masayukig has automatic API list feature on gerrit as you know. 22:52:19 <sdague> dkranz: yep, I know, but it complicates things when you need to do branch selection 22:52:24 <ken1ohmichi> sdague: will it be useful it? 22:52:25 <sdague> because tempest doesn't know that 22:52:37 <sdague> ken1ohmichi: I don't know, that's a good question 22:52:51 <dkranz> sdague: no, you would have to tell it. I will propose something when I get through the list. 22:52:52 <sdague> I think not exactly here 22:53:05 <mtreinish> ken1ohmichi: http://git.openstack.org/cgit/openstack/tempest/tree/tools/verify_tempest_config.py#n88 22:53:13 <mtreinish> that's all we need for this 22:53:20 <mtreinish> more or less 22:54:00 <sdague> ok, any other thoughts on this? 22:54:28 <mtreinish> sdague: you should probably send something out the ML announcing this change 22:54:31 <ken1ohmichi> mtreinish: thanks:) 22:54:51 <sdague> mtreinish: yeh, I want to get the first data results tomorrow before hand to make sure the jobs are even doable :) 22:54:56 <sdague> but then I will 22:55:23 <sdague> #action sdague to bring up branchless tempest on mailing list 22:55:57 <sdague> ok, so in the 5 minutes left, I want to give an early congrats to mtreinish on being QA-PTL-elect :) 22:56:11 <mtreinish> actually I was going to move onto critical reviews 22:56:14 <cyeoh> mtreinish: congrats! 22:56:17 <mtreinish> but I'm fine with that too :) 22:56:17 <sdague> officially the hat passes at the release 22:56:20 <sdague> so in 2 weeks 22:56:42 <mtreinish> thanks 22:56:48 <sdague> so I'll try to clean up a few things before abdicating to him :) 22:57:23 <sdague> ok, 4 minutes for critical reviews 22:57:25 <sdague> 3 minus 22:57:26 <mtreinish> who knows I might pull an Andrew Jackson and make you flee your house after it's official in a few hours :) 22:57:32 <mtreinish> #topic Critical Reviews 22:57:44 <mtreinish> so does anyone have any reviews they want to bring up? 22:57:47 <masayukig> https://review.openstack.org/#/c/66882/ 22:58:07 <mtreinish> #link https://review.openstack.org/#/c/66882/ 22:58:26 <mlavalle> Can core reviewers take a look at these neutron api tests: 22:58:31 <mtreinish> masayukig: ooh testscenarios fun 22:58:36 <mlavalle> https://review.openstack.org/#/c/66541 22:58:36 <mlavalle> https://review.openstack.org/#/c/65943 22:58:37 <mlavalle> https://review.openstack.org/#/c/66259 22:59:08 <mtreinish> mlavalle: sure 22:59:19 <mlavalle> thanks! 22:59:26 <sdague> ok, I think we are out of time 22:59:29 <masayukig> mtreinish: Separate the class vs Merge the class 22:59:55 <mtreinish> ok thanks everyone 23:00:04 <mtreinish> #endmeeting