22:00:28 #startmeeting qa 22:00:29 Meeting started Thu Mar 6 22:00:28 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:32 The meeting name has been set to 'qa' 22:00:47 hi, who's here today? 22:00:57 hi 22:00:58 I'm o/ 22:01:02 hi 22:01:09 o/ 22:01:15 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:01:17 o/ 22:01:19 ^^^ Today's agenda 22:02:04 well, let's get started 22:02:16 #topic Finding a volunteer to organize a bug day 22:02:38 so during last weeks meeting sdague and I looked at the bug queue and we have a lot of untriaged bugs 22:02:55 so it's probably a good idea to have another bug day before the icehouse release 22:03:04 we need a volunteer to organize the bug day 22:03:10 does anyone want to do it? 22:03:13 I can 22:03:22 maurosr: awesome 22:03:24 maurosr: great thanks 22:03:37 #action maurosr to organize bug day 22:03:44 #action maurosr to organize a bug day 22:03:50 jinx 22:03:57 heh 22:03:59 * sdague will stop using #action 22:04:01 mtreinish: There are about 40 New 22:04:01 ok well then let's move on then 22:04:02 yw guys, I've being slowly doing triage on background, but yeah, to many bugs hehe 22:04:24 dkranz: yeah and a lot more unassigned too I think 22:04:38 mtreinish: We can't assign them 22:04:38 yup, a lot more ^ 22:04:56 mtreinish: But if each core reviewer did 4-5 we would be done 22:05:06 mtreinish: Of course any one can help 22:05:12 dkranz: yeah that's part of it just to get eyes on the bugs 22:05:25 mtreinish: But if we agreed to all do this by next week we would not have to have a bug day or some one to organize it 22:05:56 dkranz: well I'm on vacation for the next few days so I can't do that :) 22:06:13 but anyway lets move on 22:06:14 mtreinish: By next Thursday? 22:06:14 so, honestly, my suggestion is not next week 22:06:41 mtreinish: Just saying I don't think we need to struggle to find some one to organize a bug day 22:06:51 dkranz: I disagree 22:06:51 sdague: ok, not next week 22:07:05 sdague: ok 22:07:09 because we've seen the bug count actually go down on bug days 22:07:14 and drift up all the rest of the time 22:07:26 getting everyone to focus motivates people a lot more 22:07:33 because you aren't slogging on it alone 22:07:36 sdague: I was making an alternaative proposal, not saying let the bugs rise 22:07:46 sdague: but o 22:07:48 k 22:08:02 dkranz: sure, I've just never seen it be effective. If it is, cool :) 22:08:14 sdague: Won't know unless we try 22:08:36 I feel like it's been tried 22:08:43 maybe I misremember 22:08:55 anyway, I'd say we have a volunteer to organize 22:08:57 and lets do that 22:09:18 ok the next topic is: 22:09:20 #topic Blueprints 22:09:33 mtreinish: #link https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests 22:09:39 so before we review the status of the high prio bp does anyone have a bp to bring up 22:09:42 andreaf: ok 22:09:50 andreaf: go ahead 22:09:56 mtreinish: ok 22:10:11 mtreinish: I moved the target to icehouse-rc now 22:10:29 mtreinish: one question I have, are we affected by the freeze as well now? 22:10:37 andreaf: no 22:10:41 mtreinish: ok 22:10:46 tempest will take features until the release 22:11:26 mtreinish: so I still hope to make good progress there until icehouse, but I need more reviews - I got some very good input from maurosr so far, but I would like a bit more to go faster 22:11:41 mtreinish: I know neutron and heat are higher prio 22:11:58 mtreinish: #link https://review.openstack.org/#/c/74387/ 22:12:08 #link https://review.openstack.org/#/c/77594/ 22:12:08 andreaf: yeh, I think this is game for rc 22:12:55 andreaf: I'll try to take a look at it soon 22:13:07 mtreinish: thanks! that's all I have on this :) 22:13:10 andreaf: we can definitely get this in by rc I think 22:13:11 andreaf: do we have an job with this set? 22:13:26 sdague: a v3 auth job? 22:13:27 so that we're using v3 for the whole thing 22:13:28 yeh 22:13:46 I don't think so yet right now it would be a mix of v2 and v3 22:14:02 because certain things don't support v3 auth yet 22:14:08 like the official clients for scenario 22:14:16 sdague: not yet, I'm covering with unit tests for now, but I could setup of some experimental job 22:14:28 andreaf: yeh, lets do an experimental job 22:14:39 then we can see what's working 22:14:58 that will be helpful I think in reviewing as well 22:15:01 sdague: ok, I'll do that 22:15:06 I think it would all work just somethings would be v2 and others v3 22:15:06 cool, thank yuo 22:15:28 ok did anyone else have a bp otherwise we'll just go down the list 22:16:10 #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests 22:16:20 ^^^ cyeoh any updates on that bp 22:16:48 should we should defer it to juno or do you think keep it for the release? 22:17:22 mtreinish: I don't much more will be done on it this cycle 22:17:22 we should ask cyeoh to split it up 22:17:39 sdague: split it up into? 22:17:42 cyeoh: can you sumarize what was done during this cycle in the blueprint 22:17:47 then we'll close it 22:17:51 and open a new one for juno 22:17:51 sdague: sure, will do 22:18:19 thanks 22:18:47 ok the next high prio bp: #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full 22:18:52 ok. Sorry got to run and drop my daughter off now 22:19:01 rossella_s: ^^^ is there any update on this one? 22:19:06 salv-orlando: ^^^ 22:19:38 mtreinish: bugs have been assigned, but I have seen no patches so far 22:20:02 perhaps because of the I-3 deadline. I hope to see activity in the next few days. 22:20:11 salv-orlando: great 22:20:18 yeh i3 is always a crazy time 22:20:18 salv-orlando: ok cool, I'll keep it as targeted for rc then 22:20:31 ok, thanks. 22:21:05 next up is: 22:21:07 #link https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration 22:21:19 I saw some patches about this earlier today 22:21:30 stevebaker: ^^^ any update on this 22:21:31 yeh, so I think we've got 2 tempest patches to land 22:21:41 I talked with stevebaker about this earlier today 22:21:59 after those 2 tempest patches, we're ready to turn on the heat-slow job 22:22:05 I have the config change up for review 22:22:39 https://review.openstack.org/#/c/78698/ 22:22:46 sdague: ok awesome 22:22:51 so I'm hoping we're going to do that this week 22:23:21 yeah that'll be great 22:23:34 ok the next up is mine 22:23:36 #link https://blueprints.launchpad.net/tempest/+spec/unit-tests 22:23:46 this one should be on track for the release 22:24:01 although I don't know if we really had set goals for where we wanted to be with coverage when we started it 22:24:19 but we're sitting around 13% coverage right now 22:24:49 but I think it will be a continual effort and welcome low hanging fruit for the future after icehouse 22:25:05 mtreinish: How are you measuring coverage? 22:25:19 dkranz: I manually run the coverage tox job 22:25:30 but we have coverage running on the post pipeline for tempest patches too 22:25:41 mtreinish: ok 22:25:41 I just can't remember where the output gets dumped 22:25:47 clarkb: ^^^ do you know 22:25:48 ? 22:25:49 mtreinish: ok but coverage to the tests themselves is ~ provided by running them 22:26:03 andreaf: yeah so the number isn't all that useful 22:26:06 that's a good point 22:26:09 andreaf: Yes, I was wondering 13% of what? 22:26:11 mtreinish: logs.openstack.org/post/firsttwocharsofsha1/sha1/mumble/mumble 22:26:19 clarkb: ok thanks 22:26:27 dkranz: all of the tempest code 22:26:40 mtreinish: From just the unit tests? 22:26:50 mtreinish: Or that plus a full tempest run 22:27:24 dkranz: yeah: (what lines gets executed from a unit test run) / (total number of lines) x 100 :) 22:27:26 is it possible to tell coverage to ignore any method called test_* ? 22:27:51 andreaf: maybe I'd have to check the coverage module options 22:28:02 I think there was a way to filter it 22:28:06 mtreinish: are we going to talk about Neutron testing? 22:28:27 mlavalle: yeah that's the next topic 22:28:35 :-) 22:28:56 ok I guess we can move on 22:29:02 #topic Neutron testing 22:29:09 mlavalle: go ahead 22:29:27 the group of people contributing to api tests have continue well engaged 22:29:46 over the past 2 weeks we have merged nine patchsets 22:30:30 afazekas gave us a big push an we have 6 patch sets that only require a second core review to merge 22:30:37 can I pimp them now? 22:31:09 sure 22:31:38 mlavalle: how many are still up for review? 22:31:38 mlavalle: sure I guess 22:31:52 https://review.openstack.org/#/c/71251 22:31:53 https://review.openstack.org/#/c/64271 22:31:53 https://review.openstack.org/#/c/66454 22:31:54 https://review.openstack.org/#/c/61118/ 22:31:54 https://review.openstack.org/#/c/66796 22:31:54 https://review.openstack.org/#/c/68626 22:32:03 I would say another 7 or 8 22:32:21 if we complete them all, we will have full api coverage at the end of icehouse 22:32:34 nice 22:32:48 mlavalle: really full coverage, that'd be pretty cool 22:32:50 there are 2 or 3 abandoned. I will ping their owners over the next few days 22:32:59 I don't think we're there with any of the other projects 22:33:25 if they are not ready to continue, I will reassign them 22:33:44 that's my update 22:33:46 mlavalle: ok is there anything else on neutron testing? 22:33:51 ok cool then we'll move on 22:33:53 thanks 22:34:07 #topic Heat testing 22:34:14 so I think we covered this during the bp section 22:34:20 sdague: unless you had something to add 22:34:27 nope, I think we covered it all 22:34:32 very excited to get that gating 22:34:44 yeah it'll be good 22:34:54 is the patch you posted before for symmetric gating everywhere? 22:35:19 yes 22:35:38 ok awesome 22:35:47 moving on the next topic is 22:35:54 #topic Bugs 22:36:19 so I don't think there is anything to discuss here right now because of the upcoming bug day 22:36:26 unless anyone had a bug they wanted to bring up 22:37:27 ok then let's move on 22:37:34 #topic Critical Reviews 22:37:46 does anyone have any reviews they would like to get some more eyes on 22:38:20 https://review.openstack.org/#/c/77823/ 22:38:29 #link https://review.openstack.org/#/c/77823/ 22:38:39 negative tests with admin 22:39:05 https://review.openstack.org/#/c/77816/ I would like to know the odds for this getting merged at all 22:39:05 mtreinish: I will review this after the meeting but some one else would be good too 22:39:25 dkranz: I'll try to take a look at it tonight too 22:39:38 #link https://review.openstack.org/#/c/77816/ 22:40:17 yfried: I don't see why it couldn't get merged as long as it runs in the gate 22:40:24 mtreinish: It's multinode 22:40:25 and it's a real scenario 22:40:47 dkranz, yfried: oh ok yeah then without proof that it works it won't get merged 22:41:11 mtreinish: how could I prove that? you mean 3rd party verify? 22:41:26 yfried: yeah at this point that would be the only way 22:41:32 yfried: yeh, if it comes in via 3rd party CI 22:41:42 is to have a 3rd party system that supports the configuration you need for the test 22:42:01 I assume there currently isn't one? 22:42:14 not that I know of 22:42:22 dkranz: what's going on at Red Hat in that way 22:42:24 Looks like it just needs two nodes running compute but we stilll don't have that 22:42:25 yfried: I don't know of one either 22:43:04 sdague: Not quite sure. This is vanilla multinode. 22:43:06 ok. tnx guys. is that a -2 to then? 22:43:29 yfried: yeah until it can be verified in some way 22:43:31 sdague: The infra guys and lifeless say simple multinode is coming upstream 22:43:44 dkranz: is there an ETA? 22:43:49 dkranz: tripleo gate is a long way off still 22:43:58 sdague: RH does not have any specific drivers that need third-party 22:44:21 dkranz: ian was posting redhat results previously 22:44:32 given that we only test on ubuntu 22:44:38 sdague: Yes, I'm not sure what is happening with that. But I can find out. 22:44:38 not sure what happened there 22:44:48 sdague: That is in progress. 22:45:29 sdague: mtreinish: I have a related topic I was unable to get on the agenda in time. I'll wait till the end of the meeting 22:45:36 yfried: ok 22:45:49 if there aren't any other reviews we can move on to the next topic 22:45:56 mtreinish: review: #link https://review.openstack.org/#/c/78221/ not really critical but it would be nice to get it approved 22:46:07 andreaf: ok 22:46:20 mtreinish: so I can do #link: https://review.openstack.org/#/c/77602/ then 22:46:25 sdague: that one's in your court 22:46:50 #topic Strategy for creating schemas for negative auto-gen (dkranz) 22:46:54 dkranz: you're up 22:46:57 which one? 22:47:11 sdague: the first one 78221 it's a devstack change 22:47:15 sdague: https://review.openstack.org/#/c/78221/ 22:47:16 oh, devstack, nice 22:47:17 So I don't have a lot more to say about that here. Jay's comment about json-home was interesting 22:47:27 I had not seen that before 22:47:36 andreaf: +A 22:47:39 thanks 22:47:40 I encourage folks to chime in on the ml discussion about this 22:47:45 dkranz: yea I've only had a quick look but it seems that ties it all together 22:47:56 yeh, goodness 22:48:10 but we still need to work out how to get all the stuff you and docs generation needs out of the code itself 22:48:18 cyeoh: Right 22:48:37 the question is if we should hold off on more nova auto-gen tests in tempest until this is worked out 22:48:50 If the nova folks are up for it I would say yes 22:49:08 dkranz: honestly, it's going to take some time 22:49:08 so can you mature the system which handles the templates without putting too much work on the template generation itself? 22:49:16 reverse-engineering the json schema from sporadic doc is a bear 22:49:27 dkranz: ok, but given the schema 22:49:31 sdague: I know. But it would save a huge amount of tempest work 22:49:39 hows all the generation of tests? 22:49:54 dkranz: yea and essentially Kenichi and his team are doing that anyway on the nova side for V3 at least. 22:49:58 sdague: There is very little work once you have the schema 22:50:14 sdague: to convert a negative tempest test file 22:50:23 sdague: so my feeling is that using introspection etc its all possible, its just finding someone who has the time to do it. 22:50:38 dkranz: is there a good example of that posted? 22:50:59 sdague: There are a few in the tree already, the ones that mark and I seeded to show that it worked 22:51:01 sdague: this? http://git.openstack.org/cgit/openstack/tempest/tree/etc/schemas/compute 22:51:22 mtreinish: Those are the schemas 22:51:23 so where are the runs where that was used for negative testing 22:51:28 mtreinish: There are tests tha use them 22:51:34 sdague: oh the other side 22:51:45 sdague: I can find the link now if you want. 22:51:52 yeh, what I'm saying is getting the schema is still in flux a bit 22:51:54 http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/v3/flavors/test_flavors_negative.py 22:52:04 sdague: Right 22:52:23 sdague: It is hard which is why I would rather wait if the nova side is really going to happen 22:52:33 sdague: Because otherwise it is all throw-away work 22:52:46 dkranz: that's fine, I just wasn't sure if anything on the test running / reporting was still needed 22:52:47 sdague: And the urgency is not that high to justify it IMO 22:53:05 sdague: No, it is all there unless other issues show up, which they might 22:53:37 dkranz: ok, so I agree, it's low priority until we have services with discovery 22:53:45 mtreinish: So we can have more discussion on ml and revisit 22:53:50 I'd say better effort spent on the clean log files 22:53:50 dkranz: so would the people who would otherwise work on making the templates in tempest be willing to work on the autogeneration in Nova? I'm happy to help with guidance, but realistically I'm not sure when I'd have time to work on it directly 22:53:52 mtreinish: That's all on that. 22:53:55 dkranz: ok 22:53:56 sdague: :) 22:54:08 dkranz: which, btw, I got general sign off 22:54:21 in the project meeting 22:54:24 sdague: cool! 22:54:32 though we should wait until after tuesday 22:54:39 which should be the end of the FFE rush 22:54:42 sdague: ok, my patch has to be rebased anyway 22:55:03 sdague: self-inflicted with the TRACE change :) 22:55:05 but if you can do the analysis on our clean log files today and propose that for wed merge, we can do that 22:55:18 sdague: ??? 22:55:30 ok well let's open the floor for the last 5 min 22:55:32 your change had the set for safe log files empty 22:55:35 #topic open discussion 22:55:40 sdague: Right now they are all assumed dirty 22:55:46 dkranz: but they aren't 22:55:56 sdague: But I could check for clean and add them if that is what you mean 22:56:02 yes, that's what I mean 22:56:06 sdague: ok, wil do 22:56:08 because they aren't all dirty 22:56:11 can I raise an issue? 22:56:16 yfried: sure go ahead 22:56:19 we are getting into HA testing and we are looking for a way to test failover (killing services and hosts) and other issues without breaking Tempest compatibility and scope. In general - Preforming operations outside the scope of the Openstack API, with direct access to the various nodes. I wanted to know if you guys have any idea of how to approach this, in a way that we could eventually (once the gate is ready) push it in 22:56:45 yfried: so you want fault injection, basically? 22:57:05 tempest can do that itself 22:57:05 sdague: to begin with. 22:57:19 Maybe it just needs a separate job 22:57:38 dkranz: not if it's outside of the openstack api (it's not blackbox then) 22:57:41 dkranz: well what we probably need is some fault injection service 22:57:42 dkranz: I would love to see how 22:58:07 yfried: Actually it is limited to what you can do with the admin api 22:58:15 dkranz: right, but that's not HA 22:58:19 yfried: But you can stop services I think 22:58:23 I see something like a grenade type script that can mess with things from the outside 22:58:24 I agree that moving forward we need something more 22:58:34 Yes 22:58:37 but I think that needs to be a fault injection service 22:58:50 sdague: a new admin api? 22:58:52 which you could deploy along side your open stack installation 22:58:57 no, a new service 22:59:12 well, yes, it would have an API 22:59:27 sdague: you mean a new project outside tempest? 22:59:43 yfried: I think the fault injection service would be outside of tempest 22:59:54 and tempest could have a dedicated directory of tests that used it 22:59:59 sdague: FIaaS? It doesn't really roll of the tongue 23:00:15 yfried: yeah it's definitely outside the scope of tempest 23:00:16 :) 23:00:25 Agreed 23:00:35 however, it's not outside the scope of QA program 23:00:44 tkammer suggested some kind of decorator based approach (which I'm not that familiar with) 23:00:55 I think another effort here would be interesting if someone wanted to take this on 23:00:59 yfried: Not sure what you mean 23:01:00 sdague: yeah it would definitely be part of the qa program 23:01:17 well we're out of time for today 23:01:18 since we are out of time, I'll post it to the ML and continue there 23:01:20 thanks everyone 23:01:23 sure 23:01:24 thanks 23:01:28 #endmeeting