22:00:28 <mtreinish> #startmeeting qa
22:00:29 <openstack> Meeting started Thu Mar  6 22:00:28 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:32 <openstack> The meeting name has been set to 'qa'
22:00:47 <mtreinish> hi, who's here today?
22:00:57 <masayukig> hi
22:00:58 <maurosr> I'm o/
22:01:02 <mlavalle> hi
22:01:09 <dkranz> o/
22:01:15 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
22:01:17 <andreaf> o/
22:01:19 <mtreinish> ^^^ Today's agenda
22:02:04 <mtreinish> well, let's get started
22:02:16 <mtreinish> #topic Finding a volunteer to organize a bug day
22:02:38 <mtreinish> so during last weeks meeting sdague and I looked at the bug queue and we have a lot of untriaged bugs
22:02:55 <mtreinish> so it's probably a good idea to have another bug day before the icehouse release
22:03:04 <mtreinish> we need a volunteer to organize the bug day
22:03:10 <mtreinish> does anyone want to do it?
22:03:13 <maurosr> I can
22:03:22 <sdague> maurosr: awesome
22:03:24 <mtreinish> maurosr: great thanks
22:03:37 <sdague> #action maurosr to organize bug day
22:03:44 <mtreinish> #action maurosr to organize a bug day
22:03:50 <sdague> jinx
22:03:57 <mtreinish> heh
22:03:59 * sdague will stop using #action
22:04:01 <dkranz> mtreinish: There are about 40 New
22:04:01 <mtreinish> ok well then let's move on then
22:04:02 <maurosr> yw guys, I've being slowly doing triage on background, but yeah, to many bugs hehe
22:04:24 <mtreinish> dkranz: yeah and a lot more unassigned too I think
22:04:38 <dkranz> mtreinish: We can't assign them
22:04:38 <maurosr> yup, a lot more ^
22:04:56 <dkranz> mtreinish: But if each core reviewer did 4-5 we would be done
22:05:06 <dkranz> mtreinish: Of course any one can help
22:05:12 <mtreinish> dkranz: yeah that's part of it just to get eyes on the bugs
22:05:25 <dkranz> mtreinish: But if we agreed to all do this by next week we would not have to have a bug day or some one to organize it
22:05:56 <mtreinish> dkranz: well I'm on vacation for the next few days so I can't do that :)
22:06:13 <mtreinish> but anyway lets move on
22:06:14 <dkranz> mtreinish: By next Thursday?
22:06:14 <sdague> so, honestly, my suggestion is not next week
22:06:41 <dkranz> mtreinish: Just saying I don't think we need to struggle to find some one to organize a bug day
22:06:51 <sdague> dkranz: I disagree
22:06:51 <dkranz> sdague: ok, not next week
22:07:05 <dkranz> sdague: ok
22:07:09 <sdague> because we've seen the bug count actually go down on bug days
22:07:14 <sdague> and drift up all the rest of the time
22:07:26 <sdague> getting everyone to focus motivates people a lot more
22:07:33 <sdague> because you aren't slogging on it alone
22:07:36 <dkranz> sdague: I was making an alternaative proposal, not saying let the bugs rise
22:07:46 <dkranz> sdague: but o
22:07:48 <dkranz> k
22:08:02 <sdague> dkranz: sure, I've just never seen it be effective. If it is, cool :)
22:08:14 <dkranz> sdague: Won't know unless we try
22:08:36 <sdague> I feel like it's been tried
22:08:43 <sdague> maybe I misremember
22:08:55 <sdague> anyway, I'd say we have a volunteer to organize
22:08:57 <sdague> and lets do that
22:09:18 <mtreinish> ok the next topic is:
22:09:20 <mtreinish> #topic Blueprints
22:09:33 <andreaf> mtreinish: #link https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests
22:09:39 <mtreinish> so before we review the status of the high prio bp does anyone have a bp to bring up
22:09:42 <mtreinish> andreaf: ok
22:09:50 <mtreinish> andreaf: go ahead
22:09:56 <andreaf> mtreinish: ok
22:10:11 <andreaf> mtreinish: I moved the target to icehouse-rc now
22:10:29 <andreaf> mtreinish: one question I have, are we affected by the freeze as well now?
22:10:37 <mtreinish> andreaf: no
22:10:41 <andreaf> mtreinish: ok
22:10:46 <mtreinish> tempest will take features until the release
22:11:26 <andreaf> mtreinish: so I still hope to make good progress there until icehouse, but I need more reviews - I got some very good input from maurosr so far, but I would like a bit more to go faster
22:11:41 <andreaf> mtreinish: I know neutron and heat are higher prio
22:11:58 <andreaf> mtreinish: #link https://review.openstack.org/#/c/74387/
22:12:08 <andreaf> #link https://review.openstack.org/#/c/77594/
22:12:08 <sdague> andreaf: yeh, I think this is game for rc
22:12:55 <mtreinish> andreaf: I'll try to take a look at it soon
22:13:07 <andreaf> mtreinish: thanks! that's all I have on this :)
22:13:10 <mtreinish> andreaf: we can definitely get this in by rc I think
22:13:11 <sdague> andreaf: do we have an job with this set?
22:13:26 <mtreinish> sdague: a v3 auth job?
22:13:27 <sdague> so that we're using v3 for the whole thing
22:13:28 <sdague> yeh
22:13:46 <mtreinish> I don't think so yet right now it would be a mix of v2 and v3
22:14:02 <mtreinish> because certain things don't support v3 auth yet
22:14:08 <mtreinish> like the official clients for scenario
22:14:16 <andreaf> sdague: not yet, I'm covering with unit tests for now, but I could setup of some experimental job
22:14:28 <sdague> andreaf: yeh, lets do an experimental job
22:14:39 <sdague> then we can see what's working
22:14:58 <sdague> that will be helpful I think in reviewing as well
22:15:01 <andreaf> sdague: ok, I'll do that
22:15:06 <mtreinish> I think it would all work just somethings would be v2 and others v3
22:15:06 <sdague> cool, thank yuo
22:15:28 <mtreinish> ok did anyone else have a bp otherwise we'll just go down the list
22:16:10 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
22:16:20 <mtreinish> ^^^ cyeoh any updates on that bp
22:16:48 <mtreinish> should we should defer it to juno or do you think keep it for the release?
22:17:22 <cyeoh> mtreinish: I don't much more will be done on it this cycle
22:17:22 <sdague> we should ask cyeoh to split it up
22:17:39 <mtreinish> sdague: split it up into?
22:17:42 <sdague> cyeoh: can you sumarize what was done during this cycle in the blueprint
22:17:47 <sdague> then we'll close it
22:17:51 <sdague> and open a new one for juno
22:17:51 <cyeoh> sdague: sure, will do
22:18:19 <sdague> thanks
22:18:47 <mtreinish> ok the next high prio bp: #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
22:18:52 <cyeoh> ok. Sorry got to run and drop my daughter off now
22:19:01 <mtreinish> rossella_s: ^^^ is there any update on this one?
22:19:06 <mtreinish> salv-orlando: ^^^
22:19:38 <salv-orlando> mtreinish: bugs have been assigned, but I have seen no patches so far
22:20:02 <salv-orlando> perhaps because of the I-3 deadline. I hope to see activity in the next few days.
22:20:11 <sdague> salv-orlando: great
22:20:18 <sdague> yeh i3 is always a crazy time
22:20:18 <mtreinish> salv-orlando: ok cool, I'll keep it as targeted for rc then
22:20:31 <salv-orlando> ok, thanks.
22:21:05 <mtreinish> next up is:
22:21:07 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration
22:21:19 <mtreinish> I saw some patches about this earlier today
22:21:30 <mtreinish> stevebaker: ^^^ any update on this
22:21:31 <sdague> yeh, so I think we've got 2 tempest patches to land
22:21:41 <sdague> I talked with stevebaker about this earlier today
22:21:59 <sdague> after those 2 tempest patches, we're ready to turn on the heat-slow job
22:22:05 <sdague> I have the config change up for review
22:22:39 <sdague> https://review.openstack.org/#/c/78698/
22:22:46 <mtreinish> sdague: ok awesome
22:22:51 <sdague> so I'm hoping we're going to do that this week
22:23:21 <mtreinish> yeah that'll be great
22:23:34 <mtreinish> ok the next up is mine
22:23:36 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/unit-tests
22:23:46 <mtreinish> this one should be on track for the release
22:24:01 <mtreinish> although I don't know if we really had set goals for where we wanted to be with coverage when we started it
22:24:19 <mtreinish> but we're sitting around 13% coverage right now
22:24:49 <mtreinish> but I think it will be a continual effort and welcome low hanging fruit for the future after icehouse
22:25:05 <dkranz> mtreinish: How are you measuring coverage?
22:25:19 <mtreinish> dkranz: I manually run the coverage tox job
22:25:30 <mtreinish> but we have coverage running on the post pipeline for tempest patches too
22:25:41 <dkranz> mtreinish: ok
22:25:41 <mtreinish> I just can't remember where the output gets dumped
22:25:47 <mtreinish> clarkb: ^^^ do you know
22:25:48 <mtreinish> ?
22:25:49 <andreaf> mtreinish: ok but coverage to the tests themselves is ~ provided by running them
22:26:03 <mtreinish> andreaf: yeah so the number isn't all that useful
22:26:06 <mtreinish> that's a good point
22:26:09 <dkranz> andreaf: Yes, I was wondering 13% of what?
22:26:11 <clarkb> mtreinish: logs.openstack.org/post/firsttwocharsofsha1/sha1/mumble/mumble
22:26:19 <mtreinish> clarkb: ok thanks
22:26:27 <mtreinish> dkranz: all of the tempest code
22:26:40 <dkranz> mtreinish: From just the unit tests?
22:26:50 <dkranz> mtreinish: Or that plus a full tempest run
22:27:24 <mtreinish> dkranz: yeah: (what lines gets executed from a unit test run) / (total number of lines) x 100 :)
22:27:26 <andreaf> is it possible to tell coverage to ignore any method called test_* ?
22:27:51 <mtreinish> andreaf: maybe I'd have to check the coverage module options
22:28:02 <mtreinish> I think there was a way to filter it
22:28:06 <mlavalle> mtreinish: are we going to talk about Neutron testing?
22:28:27 <mtreinish> mlavalle: yeah that's the next topic
22:28:35 <mlavalle> :-)
22:28:56 <mtreinish> ok I guess we can move on
22:29:02 <mtreinish> #topic Neutron testing
22:29:09 <mtreinish> mlavalle: go ahead
22:29:27 <mlavalle> the group of people contributing to api tests have continue well engaged
22:29:46 <mlavalle> over the past 2 weeks we have merged nine patchsets
22:30:30 <mlavalle> afazekas gave us a big push an we have 6 patch sets that only require a second core review to merge
22:30:37 <mlavalle> can I pimp them now?
22:31:09 <sdague> sure
22:31:38 <mtreinish> mlavalle: how many are still up for review?
22:31:38 <mtreinish> mlavalle: sure I guess
22:31:52 <mlavalle> https://review.openstack.org/#/c/71251
22:31:53 <mlavalle> https://review.openstack.org/#/c/64271
22:31:53 <mlavalle> https://review.openstack.org/#/c/66454
22:31:54 <mlavalle> https://review.openstack.org/#/c/61118/
22:31:54 <mlavalle> https://review.openstack.org/#/c/66796
22:31:54 <mlavalle> https://review.openstack.org/#/c/68626
22:32:03 <mlavalle> I would say another 7 or 8
22:32:21 <mlavalle> if we complete them all, we will have full api coverage at the end of icehouse
22:32:34 <sdague> nice
22:32:48 <mtreinish> mlavalle: really full coverage, that'd be pretty cool
22:32:50 <mlavalle> there are 2 or 3 abandoned. I will ping their owners over the next few days
22:32:59 <mtreinish> I don't think we're there with any of the other projects
22:33:25 <mlavalle> if they are not ready to continue, I will reassign them
22:33:44 <mlavalle> that's my update
22:33:46 <mtreinish> mlavalle: ok is there anything else on neutron testing?
22:33:51 <mtreinish> ok cool then we'll move on
22:33:53 <mlavalle> thanks
22:34:07 <mtreinish> #topic Heat testing
22:34:14 <mtreinish> so I think we covered this during the bp section
22:34:20 <mtreinish> sdague: unless you had something to add
22:34:27 <sdague> nope, I think we covered it all
22:34:32 <sdague> very excited to get that gating
22:34:44 <mtreinish> yeah it'll be good
22:34:54 <mtreinish> is the patch you posted before for symmetric gating everywhere?
22:35:19 <sdague> yes
22:35:38 <mtreinish> ok awesome
22:35:47 <mtreinish> moving on the next topic is
22:35:54 <mtreinish> #topic Bugs
22:36:19 <mtreinish> so I don't think there is anything to discuss here right now because of the upcoming bug day
22:36:26 <mtreinish> unless anyone had a bug they wanted to bring up
22:37:27 <mtreinish> ok then let's move on
22:37:34 <mtreinish> #topic Critical Reviews
22:37:46 <mtreinish> does anyone have any reviews they would like to get some more eyes on
22:38:20 <dkranz> https://review.openstack.org/#/c/77823/
22:38:29 <mtreinish> #link https://review.openstack.org/#/c/77823/
22:38:39 <dkranz> negative tests with admin
22:39:05 <yfried> https://review.openstack.org/#/c/77816/ I would like to know the odds for this getting merged at all
22:39:05 <dkranz> mtreinish: I will review this after the meeting but some one else would be good too
22:39:25 <mtreinish> dkranz: I'll try to take a look at it tonight too
22:39:38 <mtreinish> #link https://review.openstack.org/#/c/77816/
22:40:17 <mtreinish> yfried: I don't see why it couldn't get merged as long as it runs in the gate
22:40:24 <dkranz> mtreinish: It's multinode
22:40:25 <mtreinish> and it's a real scenario
22:40:47 <mtreinish> dkranz, yfried: oh ok yeah then without proof that it works it won't get merged
22:41:11 <yfried> mtreinish: how could I prove that? you mean 3rd party verify?
22:41:26 <mtreinish> yfried: yeah at this point that would be the only way
22:41:32 <sdague> yfried: yeh, if it comes in via 3rd party CI
22:41:42 <mtreinish> is to have a 3rd party system that supports the configuration you need for the test
22:42:01 <yfried> I assume there currently isn't one?
22:42:14 <sdague> not that I know of
22:42:22 <sdague> dkranz: what's going on at Red Hat in that way
22:42:24 <dkranz> Looks like it just needs two nodes running compute but we stilll don't have that
22:42:25 <mtreinish> yfried: I don't know of one either
22:43:04 <dkranz> sdague: Not quite sure. This is vanilla multinode.
22:43:06 <yfried> ok. tnx guys. is that a -2 to then?
22:43:29 <mtreinish> yfried: yeah until it can be verified in some way
22:43:31 <dkranz> sdague: The infra guys and lifeless say simple multinode is coming upstream
22:43:44 <yfried> dkranz: is there an ETA?
22:43:49 <sdague> dkranz: tripleo gate is a long way off still
22:43:58 <dkranz> sdague: RH does not have any specific drivers that need third-party
22:44:21 <sdague> dkranz: ian was posting redhat results previously
22:44:32 <sdague> given that we only test on ubuntu
22:44:38 <dkranz> sdague: Yes, I'm not sure what is happening with that. But I can find out.
22:44:38 <sdague> not sure what happened there
22:44:48 <dkranz> sdague: That is in progress.
22:45:29 <yfried> sdague: mtreinish: I have a related topic I was unable to get on the agenda in time. I'll wait till the end of the meeting
22:45:36 <mtreinish> yfried: ok
22:45:49 <mtreinish> if there aren't any other reviews we can move on to the next topic
22:45:56 <andreaf> mtreinish: review: #link https://review.openstack.org/#/c/78221/ not really critical but it would be nice to get it approved
22:46:07 <mtreinish> andreaf: ok
22:46:20 <andreaf> mtreinish: so I can do #link: https://review.openstack.org/#/c/77602/ then
22:46:25 <mtreinish> sdague: that one's in your court
22:46:50 <mtreinish> #topic Strategy for creating schemas for negative auto-gen (dkranz)
22:46:54 <mtreinish> dkranz: you're up
22:46:57 <sdague> which one?
22:47:11 <mtreinish> sdague: the first one 78221 it's a devstack change
22:47:15 <andreaf> sdague: https://review.openstack.org/#/c/78221/
22:47:16 <sdague> oh, devstack, nice
22:47:17 <dkranz> So I don't have a lot more to say about that here. Jay's comment about json-home was interesting
22:47:27 <dkranz> I had not seen that before
22:47:36 <sdague> andreaf: +A
22:47:39 <sdague> thanks
22:47:40 <dkranz> I encourage folks to chime in on the ml discussion about this
22:47:45 <cyeoh> dkranz: yea I've only had a quick look but it seems that ties it all together
22:47:56 <sdague> yeh, goodness
22:48:10 <cyeoh> but we still need to work out how to get all the stuff you and docs generation needs out of the code itself
22:48:18 <dkranz> cyeoh: Right
22:48:37 <dkranz> the question is if we should hold off on more nova auto-gen tests in tempest until this is worked out
22:48:50 <dkranz> If the nova folks are up for it I would say yes
22:49:08 <sdague> dkranz: honestly, it's going to take some time
22:49:08 <cyeoh> so can you mature the system which handles the templates without putting too much work on the template generation itself?
22:49:16 <dkranz> reverse-engineering the json schema from sporadic doc is a bear
22:49:27 <sdague> dkranz: ok, but given the schema
22:49:31 <dkranz> sdague: I know. But it would save a huge amount of tempest work
22:49:39 <sdague> hows all the generation of tests?
22:49:54 <cyeoh> dkranz: yea and essentially Kenichi and his team are doing that anyway on the nova side for V3 at least.
22:49:58 <dkranz> sdague: There is very little work once you have the schema
22:50:14 <dkranz> sdague: to convert a negative tempest test file
22:50:23 <cyeoh> sdague: so my feeling is that using introspection etc its all possible, its just finding someone who has the time to do it.
22:50:38 <sdague> dkranz: is there a good example of that posted?
22:50:59 <dkranz> sdague: There are a few in the tree already, the ones that mark and I seeded to show that it worked
22:51:01 <mtreinish> sdague: this? http://git.openstack.org/cgit/openstack/tempest/tree/etc/schemas/compute
22:51:22 <dkranz> mtreinish: Those are the schemas
22:51:23 <sdague> so where are the runs where that was used for negative testing
22:51:28 <dkranz> mtreinish: There are tests tha use them
22:51:34 <mtreinish> sdague: oh the other side
22:51:45 <dkranz> sdague: I can find the link now if you want.
22:51:52 <sdague> yeh, what I'm saying is getting the schema is still in flux a bit
22:51:54 <mtreinish> http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/v3/flavors/test_flavors_negative.py
22:52:04 <dkranz> sdague: Right
22:52:23 <dkranz> sdague: It is hard which is why I would rather wait if the nova side is really going to happen
22:52:33 <dkranz> sdague: Because otherwise it is all throw-away work
22:52:46 <sdague> dkranz: that's fine, I just wasn't sure if anything on the test running / reporting was still needed
22:52:47 <dkranz> sdague: And the urgency is not that high to justify it IMO
22:53:05 <dkranz> sdague: No, it is all there unless other issues show up, which they might
22:53:37 <sdague> dkranz: ok, so I agree, it's low priority until we have services with discovery
22:53:45 <dkranz> mtreinish: So we can have more discussion on ml and revisit
22:53:50 <sdague> I'd say better effort spent on the clean log files
22:53:50 <cyeoh> dkranz: so would the people who would otherwise work on making the templates in tempest be willing to work on the autogeneration in Nova? I'm happy to help with guidance, but realistically I'm not sure when I'd have time to work on it directly
22:53:52 <dkranz> mtreinish: That's all on that.
22:53:55 <mtreinish> dkranz: ok
22:53:56 <dkranz> sdague: :)
22:54:08 <sdague> dkranz: which, btw, I got general sign off
22:54:21 <sdague> in the project meeting
22:54:24 <dkranz> sdague: cool!
22:54:32 <sdague> though we should wait until after tuesday
22:54:39 <sdague> which should be the end of the FFE rush
22:54:42 <dkranz> sdague: ok, my patch has to be rebased anyway
22:55:03 <dkranz> sdague: self-inflicted with the TRACE change :)
22:55:05 <sdague> but if you can do the analysis on our clean log files today and propose that for wed merge, we can do that
22:55:18 <dkranz> sdague: ???
22:55:30 <mtreinish> ok well let's open the floor for the last 5 min
22:55:32 <sdague> your change had the set for safe log files empty
22:55:35 <mtreinish> #topic open discussion
22:55:40 <dkranz> sdague: Right now they are all assumed dirty
22:55:46 <sdague> dkranz: but they aren't
22:55:56 <dkranz> sdague: But I could check for clean and add them if that is what you mean
22:56:02 <sdague> yes, that's what I mean
22:56:06 <dkranz> sdague: ok, wil do
22:56:08 <sdague> because they aren't all dirty
22:56:11 <yfried> can I raise an issue?
22:56:16 <mtreinish> yfried: sure go ahead
22:56:19 <yfried> we are getting into HA testing and we are looking for a way to test failover (killing services and hosts) and other issues without breaking Tempest compatibility and scope. In general - Preforming operations outside the scope of the Openstack API, with direct access to the various nodes. I wanted to know if you guys have any idea of how to approach this, in a way that we could eventually (once the gate is ready) push it in
22:56:45 <sdague> yfried: so you want fault injection, basically?
22:57:05 <dkranz> tempest can do that itself
22:57:05 <yfried> sdague: to begin with.
22:57:19 <dkranz> Maybe it just needs a separate job
22:57:38 <mtreinish> dkranz: not if it's outside of the openstack api (it's not blackbox then)
22:57:41 <sdague> dkranz: well what we probably need is some fault injection service
22:57:42 <yfried> dkranz: I would love to see how
22:58:07 <dkranz> yfried: Actually it is limited to what you can do with the admin api
22:58:15 <sdague> dkranz: right, but that's not HA
22:58:19 <dkranz> yfried: But you can stop services I think
22:58:23 <mtreinish> I see something like a grenade type script that can mess with things from the outside
22:58:24 <sdague> I agree that moving forward we need something more
22:58:34 <dkranz> Yes
22:58:37 <sdague> but I think that needs to be a fault injection service
22:58:50 <dkranz> sdague: a new admin api?
22:58:52 <sdague> which you could deploy along side your open stack installation
22:58:57 <sdague> no, a new service
22:59:12 <sdague> well, yes, it would have an API
22:59:27 <yfried> sdague: you mean a new project outside tempest?
22:59:43 <sdague> yfried: I think the fault injection service would be outside of tempest
22:59:54 <sdague> and tempest could have a dedicated directory of tests that used it
22:59:59 <mtreinish> sdague: FIaaS? It doesn't really roll of the tongue
23:00:15 <mtreinish> yfried: yeah it's definitely outside the scope of tempest
23:00:16 <sdague> :)
23:00:25 <dkranz> Agreed
23:00:35 <sdague> however, it's not outside the scope of QA program
23:00:44 <yfried> tkammer suggested some kind of decorator based approach (which I'm not that familiar with)
23:00:55 <sdague> I think another effort here would be interesting if someone wanted to take this on
23:00:59 <dkranz> yfried: Not sure what you mean
23:01:00 <mtreinish> sdague: yeah it would definitely be part of the qa program
23:01:17 <mtreinish> well we're out of time for today
23:01:18 <yfried> since we are out of time, I'll post it to the ML and continue there
23:01:20 <mtreinish> thanks everyone
23:01:23 <sdague> sure
23:01:24 <sdague> thanks
23:01:28 <mtreinish> #endmeeting