22:00:14 #startmeeting qa 22:00:14 Meeting started Thu Apr 3 22:00:14 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:18 The meeting name has been set to 'qa' 22:00:26 hi who do we have here today? 22:00:28 o/ 22:00:32 o/ 22:00:34 hi 22:00:51 o/ 22:00:52 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:00:58 ^^^ Today's agenda 22:01:05 o/ 22:01:42 ok lets get started 22:01:57 #topic Summit sessions (mtreinish) 22:02:00 o/ 22:02:16 so I started an etherpad to track design summit proposals: 22:02:20 #link https://etherpad.openstack.org/p/Juno-QA-design-summit-topics 22:02:42 I thought it would be good to do what we did last cycle and have a discussion in the meeting about the sessions 22:03:02 what else has been submitted into the tool? 22:03:03 so if people had ideas for a summit session if you could put them in the etherpad 22:03:24 sdague: right now there are only 2 22:03:36 probably worth transposing them to the etherpad 22:03:45 "A case for project level functional testing" and "Rally & Tempest integration (future steps)" 22:04:00 the first one sounds like an overlap of the one dkranz put in 22:04:05 sdague: It is 22:04:16 sdague: I didn't know about the others 22:04:18 sdague: My bad 22:04:23 dkranz: no worries 22:04:55 I guess we should talk about how we can reduce the code overlap between v2 and v3 tests, but I doubt we need whole session on that 22:04:57 dkranz: http://summit.openstack.org/cfp/details/134 22:05:25 cyeoh: well, a refactoring for maintainability session might be good 22:05:27 include that 22:05:30 cyeoh: yeah probably not, but it could be part of a larger session 22:05:38 plus other items people have 22:05:58 I still kind of want to do the nice skip decorators, but that's back burner 22:06:21 so anyway if people have ideas if they can put them on the etherpad and we'll go through it in a couple of weeks during the meeting 22:06:30 yeh, probably plan for 3 weeks out 22:06:38 as decision time 22:06:43 * mtreinish needs to look at a calendar 22:06:58 ok that's all I had to talk about on this for now 22:07:01 so let's move on 22:07:10 #topic Blueprints 22:07:21 so sdague why don't you start of on the specs stuff 22:07:25 sure 22:07:28 #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z 22:07:37 there are the open specs we currently have 22:08:22 currently all of qa-core has +2 rights, which I think is right 22:08:32 though we should probably by convention let the PTL do +A 22:08:42 just to make sure they are in on the vote 22:08:45 or ack it 22:09:10 I think right now we have one that's fully ready 22:09:16 https://review.openstack.org/#/c/83264/ 22:09:29 and I went through the rest of them and provided some feedback on additions 22:09:34 +1 for the +A policy 22:09:45 anyone have objectsions to https://review.openstack.org/#/c/83264/ ? 22:09:58 sdague: I hope not it's mostly implemented :) 22:10:05 otherwise I'll land that one as our first one 22:10:17 ok, done 22:10:18 sdague: No, but should this include an 'admin' tag or is that a different bp? 22:10:30 dkranz: this is just the service tagging 22:10:39 mtreinish: ok 22:10:39 like compute, image, etc... 22:10:50 dkranz: I think admin clean is a different qa-spec 22:10:58 can you do a draft on that? 22:11:00 which all thats left on is going through and finding where we need to tag things 22:11:09 I know you were very interested in that yesterday 22:11:39 sdague: Yes 22:12:03 re: 83264 can you do a hacking check to see if someone does add a service name which does exist in the path 22:12:21 #action dkranz to draft no admin qa-spec 22:12:23 and flag an error if it occurs? Just to save inconsistencies creeping in accidentally 22:12:25 cyeoh: was that not in there. I intend to it came up during the spec review 22:12:39 mtreinish: cool! 22:12:41 or I thought it did 22:12:59 well, if not, we can always do an ammendment :) 22:13:14 no it's in there L52 22:13:30 it's just not a work item... 22:13:36 cyeoh, dkranz: would you guys mind doing a run through the existing specs and providing comments as well 22:13:49 I'd like to get us used to this model of making sure we have bases covered 22:13:55 sdague: Will do 22:14:00 sdague: ok 22:14:01 fortunately we've only got a few, so it's pretty quick 22:14:35 I'm also going get the infra side setup soon 22:14:37 #action sdague to reset all non high blueprints, point people at qa-specs 22:14:45 so we'll publish the specs somewhere 22:14:49 when they get merged 22:14:56 mtreinish: I'm not sure why 22:15:00 it looks pretty 22:15:04 I really feel like the specs are ok in git 22:15:20 because the blueprint is really the entry point 22:15:28 having read through a few long nova ones, I think its easier to read in processed html 22:15:35 cyeoh: ok 22:15:50 then I'll defer 22:16:19 sdague: ok is there anything else to discuss on the specs? 22:16:27 only one last thing 22:16:50 I feel like we should have ~4 specs landed as examples before we open this up to the wide world via ML 22:17:09 so will hold off a few 22:17:24 ok, done, we can move on 22:17:28 that seems fair 22:17:47 ok then let's discuss bps in progress 22:17:55 zhikunliu: you put something on the agenda 22:18:00 yes 22:18:03 https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests 22:18:04 go ahead 22:18:24 #link https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests 22:18:56 we have already create basic cinder v2 tests. Not sure if we should continue to port v1 tests to v2 like nova v2 to v3 22:19:19 zhikunliu: how much duplication is there between v1 and v2? 22:19:26 we want to have good coverage for both 22:20:15 about 10 test files 22:20:22 cyeoh and ken1ohmichi have been more plugged in with the nova v2 v3 stuff so they may have some advice 22:20:52 OK, I will see cinder API tests. 22:21:24 thank you! 22:21:39 np:) 22:21:46 zhikunliu: so we want to exercise both the v1 and v2 apis. It's just a question of we go about implementing that. Inheritance vs 2 copies for example 22:21:52 yea I guess it comes down to how similar the APIs look - we tried along the way but weren't that succesfull, but ken1ohmichi has some examples of a direction we'll probably take 22:22:13 yeh, the nova v2 api is exceptionally odd though 22:22:20 maybe we can do better here 22:22:32 yes, the sample is https://review.openstack.org/#/c/78137/ 22:22:55 so I would honestly really encourage this to take place in a qa-spec - https://github.com/openstack/qa-specs because I think that provides a really good way of reviewing the approach 22:23:07 and then we can all be agreed on approach and just review for correctness 22:23:26 I got it. I will propose it to qa-specs 22:23:40 as the one of the samples:) 22:23:43 +1, though I don't think it hurts to have a bit of POC code to point to 22:23:53 ken1ohmichi: great 22:23:56 cyeoh: agreed 22:24:15 zhikunliu: ok is there anything else on that bp? 22:24:24 no, thanks... 22:24:35 ok does anyone have any other bps they'd like to bring up? 22:25:14 if not, I'd actually like to jump to the tempest branch discussion 22:25:32 because I want to make sure we give that enough time, for folks that we're in -qa channel yesterday to see it 22:25:36 sdague: I think I got through classifying about half of the failures 22:25:40 ok then let's move onto that 22:25:44 #topic Tempest branching strategy 22:26:14 sdague: you've got the floor 22:26:30 so background 22:26:33 #link http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2014-04-02.log 22:26:42 starting at about 2014-04-02T13:29:08 22:27:04 a question came up about the state of tempest as a validation tool for the openstack brand 22:27:34 and out of that conversation I asked the question: does tempest really need branches? 22:28:00 because it *should* be testing an API that is invariant between releases 22:28:12 sdague: I think it needs versions, not branches, like client libraries. 22:28:13 or changed in ways that are detectable/flaggable 22:28:24 dkranz: sure, lets get to that in a sec 22:28:56 we have 2 weeks before we would normally set the stable/icehouse branch 22:29:04 so now is the time to decide if maybe we don't 22:29:25 and use master tempest on stable/icehouse and master going forward, and try to get tempest master to work against stable/havana 22:29:37 so I'm not saying I'm opposed to it, but my concern is if we branch of the overhead of adding new tests 22:30:06 having to add them to multiple branches if we want to use them as a validation tool 22:30:08 cyeoh: vs. the cost of backporting them? 22:30:27 because thinks like defcore/refstack are going to lag 22:30:38 at icehouse summit they are going to put out a havana based tool 22:30:43 which vendors will want to fix 22:31:14 they are going to want to fix the tool? 22:31:28 cyeoh: The tool is going to run tempest 22:31:31 I am sure they will object to some results 22:31:43 ok 22:31:53 and they may be correct in tempest fixes that should land, and then backport 22:32:36 the reason we set a branch today is because until this release we didn't have a feature enable mechanism 22:32:47 so branch was a giant feature flag 22:32:54 sdague: well we did but it was a mess 22:32:57 havana nova supports X 22:33:06 icehouse nova support X + Y 22:33:29 however the feature flag model should mean that new extensions just come in under a feature flag, for instance 22:33:47 sdague: so the prereq for this on devstack as I said yesterday 22:33:52 sdague: The big advantage of running master is that adding tests always lags the release 22:33:52 correct 22:34:01 we have to make sure we lock the feature set in the devstack config for stable/icehouse 22:34:15 sdague: RIght not there are many tests in master that can and should run when testing havana 22:34:39 sdague: And many that don't due to questionable api changes 22:34:46 dkranz: sure 22:34:48 so just to clarify for me how this would work - if we add Nova API V2 tests during Juno, what would we have to do to ensure they are added for both Juno Nova API testing as well as say Icehouse 22:35:17 tempest master would gate on master and stable/icehouse of all the server projects 22:35:17 cyeoh: but they should given the stable api :) 22:35:21 cyeoh: I would say they should run on both by default 22:35:38 dkranz: yep that's what I'd like the default to be 22:35:45 mtreinish: yea, they *should* :-) 22:35:45 I've been working on devstack gate branch overrides that would let us specify that behavior 22:35:52 cyeoh: And if a test doesn't run due to a havana bug that was fixed iceshouse we tag it as >havana 22:36:00 cyeoh: so it will only be 1 add to tempest 22:36:06 and it will gate on both branches 22:36:12 so you know if it works or not 22:36:22 sdague: ok, sounds great then :-) 22:36:30 sdague: So are we going to double the gate jobs ? 22:36:37 dkranz: we have to 22:36:38 for tempest 22:36:48 sdague: Actually triple since havana + icehouse + master 22:36:55 sdague: That is my biggest concern with this approach 22:36:58 dkranz: this will only be for icehouse forward 22:37:02 it's only on tempest 22:37:12 sdague: Ah, ok 22:37:19 mtreinish: I think getting havana over is still on the table 22:37:23 depending on how much work it is 22:37:32 it is but the discussion now is about what we do on the 17th 22:37:41 we still have the havana branch whether we like it or not 22:37:46 right, for the 17th, it would be just icehouse + master 22:37:46 mtreinish: I don't see why we have to commit 22:37:51 agreed 22:38:17 I'll put the havana jobs on nv, and if we get it working, we'll delete the havana branch, and win 22:38:23 if not 22:38:28 it ages out in 5 months 22:38:54 sdague: But it can't if refstack is using it 22:39:04 sdague: I mean can't age out 22:39:09 havana still stops being supported in 5 months 22:39:15 dkranz: havana goes eol in 5 months 22:39:17 openstack policy hasn't changed in that regard 22:39:21 we just did it for grizzly 22:39:48 sdague: That doesn't mean refstack will stop using it. Have you told them this intent? 22:39:57 sdague: Cause it makes no sense from their standpoint 22:40:09 sdague: Of course it could just be cloned into a refstack repo at that point. 22:40:26 dkranz: honestly, I'm not policing them, I have lots of other things to do :) 22:40:36 sdague: RIght :) 22:40:53 sdague: Based on my results so far I think it is very doable to get master to work with havana 22:40:58 ken1ohmichi, masayukig any thoughts on this one? I'd like to know if there are other problems we haven't thought about yet 22:41:02 dkranz: great 22:41:14 sdague: We just need a tagging strategy for the breaks due to bug fixes in icehouse and api changes 22:41:16 I should have gate jobs running nv tomorrow that will let us do that 22:41:39 dkranz: example? 22:41:54 sdague: https://etherpad.openstack.org/p/master-against-havana-errors 22:42:26 honestly, I am not familar with refstack. 22:42:33 dkranz: I was wondering about "tagging strategy" 22:42:51 ken1ohmichi: refstack is basically a website that some foundation folks will run where you can submit tempest tests 22:42:52 ken1ohmichi: It is the "use tempest to gate use of OpenStack trademark" 22:43:01 test results 22:43:04 for your product 22:43:16 as a piece of trademark use policy 22:43:28 they are still working out all the trademark details 22:43:35 sdague: about tagging strategy, I think we should have a tag saying a test is enabled as of some version 22:43:49 running a subset of tempest successfully will be a part of what you need to do to get the trademark 22:43:50 dkranz: that should be handled by config 22:44:00 right, I agree with mtreinish 22:44:05 oh, thanks. I got it little. 22:44:07 sdague: one thing that came to my mind is the maintenance burden, like in this situation we cant remove nova's xml tests 22:44:20 mtreinish: how? some of the tests have different havana/icehouse behavior due to api change 22:44:20 (not sure if you guys touched this matter, lost my connection) 22:44:27 sdague: Does refstack still uses grizzly? 22:44:28 maurosr: that's true, at least not for a while 22:44:38 masayukig: they are building on havana atm 22:44:45 from what I understand 22:45:04 dkranz: well those types of changes will be blocked by tempest in the first place 22:45:32 it really only becomes a problem for when we add new test coverage 22:45:45 mtreinish: which we did in icehouse 22:45:47 since api changes are not supposed to happen I guess that is a 'good break' 22:45:59 mtreinish: And the tests were not blocked. We changed them. 22:46:08 mtreinish: YOu can see examples in that etherpad 22:46:10 it's something that maybe cant work planty right now, but will be good at long term 22:46:11 dkranz: yes because we default everything on everywhere 22:46:24 but if we have a static set of what's expected in devstack for icehouse 22:46:30 then those new tests won't run 22:46:41 dkranz: there are definitely details here on what the 2 step is, but I think those are doable 22:46:50 mtreinish: But why shouldn't a new test for a stable API in a previous version run? 22:47:04 against the old version 22:47:14 mtreinish: I agree we can do better icehouse->juno 22:47:26 dkranz: it should but we'd need a devstack change to add it to the stable job (it's just a shift in the 2 step) 22:47:32 so I think what we probably want to do is after stable/icehouse on devstack is set 22:47:52 we generate the full extension list and hardcode it in devstack stable/icehouse 22:47:55 but it floats on master 22:48:01 so stuff is auto added 22:48:07 sdague: exactly 22:48:19 but that's actually after the 17th 22:48:36 yeah it would have to be because we need the stable branch on devstack 22:48:38 sdague: That's fine but half the problem. The other part is stuff that actually changed, or that works in icehouse but not havana due to an unfixed bug in havana 22:48:48 dkranz: sure 22:48:56 sdague: At a minimum we could just skip all the fails I found for havana. 22:49:08 sdague: That would be a better test than what is on stable/havana now. 22:49:19 well, I don't want to branch skip 22:49:24 I think that logic gets nutty 22:49:36 so we either fix it, feature flag it, or the job doesn't vote 22:49:58 sdague: Why? We just classify the test as belonging to a new feature. Just a funny one. 22:50:18 because we're already going to have 200 features 22:50:23 that are real ones 22:50:28 sdague: And this would be one more. 22:50:46 I think if we invent funny ones that we can't double check on real extension lists we're going to get it wrong 22:50:57 there is a big who watches the watcher thing here 22:51:02 sdague: Anyway, the goal is to run master against havana with as many tests as we can get to work. 22:51:10 sdague: We need to figure the details but not right now. 22:51:14 sure 22:51:21 sdague: the extension list means API list? 22:51:29 ken1ohmichi: yes 22:51:43 ken1ohmichi: yeah it's the api_extensions config option 22:51:57 sdague: What I am talking about is more like bug skipping than feature config. 22:52:05 sdague: ok, masayukig has automatic API list feature on gerrit as you know. 22:52:19 dkranz: yep, I know, but it complicates things when you need to do branch selection 22:52:24 sdague: will it be useful it? 22:52:25 because tempest doesn't know that 22:52:37 ken1ohmichi: I don't know, that's a good question 22:52:51 sdague: no, you would have to tell it. I will propose something when I get through the list. 22:52:52 I think not exactly here 22:53:05 ken1ohmichi: http://git.openstack.org/cgit/openstack/tempest/tree/tools/verify_tempest_config.py#n88 22:53:13 that's all we need for this 22:53:20 more or less 22:54:00 ok, any other thoughts on this? 22:54:28 sdague: you should probably send something out the ML announcing this change 22:54:31 mtreinish: thanks:) 22:54:51 mtreinish: yeh, I want to get the first data results tomorrow before hand to make sure the jobs are even doable :) 22:54:56 but then I will 22:55:23 #action sdague to bring up branchless tempest on mailing list 22:55:57 ok, so in the 5 minutes left, I want to give an early congrats to mtreinish on being QA-PTL-elect :) 22:56:11 actually I was going to move onto critical reviews 22:56:14 mtreinish: congrats! 22:56:17 but I'm fine with that too :) 22:56:17 officially the hat passes at the release 22:56:20 so in 2 weeks 22:56:42 thanks 22:56:48 so I'll try to clean up a few things before abdicating to him :) 22:57:23 ok, 4 minutes for critical reviews 22:57:25 3 minus 22:57:26 who knows I might pull an Andrew Jackson and make you flee your house after it's official in a few hours :) 22:57:32 #topic Critical Reviews 22:57:44 so does anyone have any reviews they want to bring up? 22:57:47 https://review.openstack.org/#/c/66882/ 22:58:07 #link https://review.openstack.org/#/c/66882/ 22:58:26 Can core reviewers take a look at these neutron api tests: 22:58:31 masayukig: ooh testscenarios fun 22:58:36 https://review.openstack.org/#/c/66541 22:58:36 https://review.openstack.org/#/c/65943 22:58:37 https://review.openstack.org/#/c/66259 22:59:08 mlavalle: sure 22:59:19 thanks! 22:59:26 ok, I think we are out of time 22:59:29 mtreinish: Separate the class vs Merge the class 22:59:55 ok thanks everyone 23:00:04 #endmeeting