22:00:14 <mtreinish> #startmeeting qa 22:00:15 <openstack> Meeting started Thu Feb 6 22:00:14 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:20 <openstack> The meeting name has been set to 'qa' 22:00:32 <mtreinish> hi who do we have today? 22:00:33 <dkranz> Here 22:00:36 <masayukig> o/ 22:00:39 <rahmu> hello 22:00:43 <mlavalle> hello 22:00:59 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:01:04 <mtreinish> ^^^ today's agenda 22:01:12 <ken1ohmichi> hi 22:01:21 <boris-42_> hi all 22:01:22 <boris-42_> =) 22:01:29 <afazekas> hi 22:01:38 <boris-42_> how are you guys?) 22:01:43 <mtreinish> let's dive into it 22:01:54 <mtreinish> #topic Blueprints 22:02:04 <mtreinish> does anyone have a blueprint that needs attention? 22:02:14 <mtreinish> or a status update on an open blueprint? 22:03:13 <dkranz> mtreinish: no :) 22:03:26 <mtreinish> heh yeah I guess let's move on to the next topic 22:03:38 <mtreinish> #topic Neutron testing 22:03:50 <mtreinish> mlavalle: any updates on the status of things with neutron? 22:04:05 * boris-42_ enikanorov__ ping 22:04:06 <mlavalle> api tests development has continued 22:04:19 <enikanorov__> boris-42_: queque 22:04:43 <mlavalle> lot's of good contributions. We have almost a 100% coverage of the gap identified in the etherpad 22:04:57 <enikanorov__> boris-42_: greate that you've just woke me up 22:04:58 <mtreinish> oh wow are they all up for review? 22:05:18 <mlavalle> they are all in some part of the revuew cycle 22:05:27 <enikanorov__> i wanted to ask tempest cores to look at the patch we want for quite a long time: 22:05:31 <mlavalle> I am doing about 4 reviews a day 22:05:44 <enikanorov__> oh 22:05:48 <enikanorov__> i see it got approved 22:05:51 <enikanorov__> https://review.openstack.org/#/c/58697/ 22:05:59 <enikanorov__> thanks! 22:06:06 <mtreinish> mlavalle: ok and is the neutron gate stablized so we can start pushing things through? 22:06:08 <masayukig> yeah 22:06:14 <dkranz> enikanorov__: I was a little concerned about the runtime 22:06:28 <dkranz> mtreinish: It adds 45 seconds to neutron run. What should we do about this? 22:06:39 <mlavalle> salv-orlando reported good progress this past Monday 22:06:45 <mtreinish> dkranz: the neutron run isn't that much of a concern on the time budget 22:06:48 <dkranz> mtreinish: We said we would focus on scenarios but by nature they can take some time 22:06:50 <mtreinish> because they only run smoke 22:06:57 <mlavalle> I understand they are still stabilizing this week 22:07:02 <enikanorov__> dkranz: it spanws a vm. i guess we could setup a backend on the host itself 22:07:11 <dkranz> mtreinish: Yes, but they are only running a few minutes faster... 22:07:12 <enikanorov__> and just add route to the host from the tenant network 22:07:15 <mtreinish> dkranz: it'll be an issue long term but for right now it's ok 22:07:34 <dkranz> mtreinish: sdague wanted us to watch out for this 22:07:37 <mlavalle> I will ping him again today or tomorrow and pass the status in the qa channel 22:07:43 <mtreinish> mlavalle: ok cool 22:07:53 <mtreinish> I know we merged the neutron api tenant isolation patch 22:07:56 <dkranz> mtreinish: ok, I will stop worrying and love the bomb 22:08:07 <mtreinish> which previously broke things 22:08:23 <mlavalle> we've made good progress getting people contributing and I don't want to loose their enthusiasm 22:08:33 <mtreinish> dkranz: yeah sdague just said we need to be careful with what we merge to try and control the overall runtime 22:08:57 <mlavalle> that's all I have 22:09:06 <mtreinish> dkranz: if it becomes an issue we can just start tagging things as slow 22:09:21 <mtreinish> and add a new nonheat slow job 22:09:30 <dkranz> mtreinish: True 22:10:06 <mtreinish> ok does anyone have anything else to bring up about neutron testing? 22:10:44 <mtreinish> ok then let's move on 22:10:55 <mtreinish> #topic When can we enable failing jobs with bogus log ERRORs (dkranz) 22:11:02 <mtreinish> dkranz: you're up 22:11:15 <dkranz> mtreinish: I put in a lot of work to get this feature in. 22:11:28 <dkranz> mtreinish: It got turned off due to unstable gate 22:11:37 <dkranz> Now bugs are creeping back in 22:11:50 <dkranz> That are ignored because they just show up in a log that no one looks at. 22:12:12 <mtreinish> dkranz: yeah it was causing a lot of nondeterministic failures at a time when things were already really unstable 22:12:13 <dkranz> sdague made a comment that we could turn this back on after icehouse-2 22:12:21 <dkranz> mtreinish: which is now 22:12:34 <mtreinish> dkranz: we can't do that now because of the oslo.messaging errors 22:12:35 <dkranz> mtreinish: but now we have all these errors again 22:12:40 <mtreinish> every run will fail 22:12:50 <dkranz> mtreinish: right, so how do we get that fixed? 22:12:52 <rahmu> dkranz: mtreinish: can you please briefly explain what this feature is about? 22:13:00 <dkranz> rahmu: ok 22:13:26 <mtreinish> rahmu: sure, dkranz wrote a script that goes through all the service logs and prints out the error messages after a gate run 22:13:38 <mtreinish> it used to fail the job if there was an error in any of the logs 22:13:44 <dkranz> rahmu: There are a lot of bugs that let tests seem to pass even thought something is screwed up 22:14:10 <dkranz> mtreinish: is fixing the oslo thing a priority for any one? 22:14:40 <mtreinish> I know sdague brought it up on the ML, but I haven't been paying too much attention to it 22:14:55 <rahmu> mtreinish: dkranz: I understand. Thanks :) 22:15:03 <dkranz> The "lot of work" was not to write the script but to make sure all errors existing converged in a whitelist 22:15:21 <dkranz> A task we are stuck with again 22:15:38 <mtreinish> dkranz: yeah I understand, another idea that sdague and I were throwing around was to split the log checking into a separate bot 22:16:24 <dkranz> mtreinish: You mean make it not part of the gate? 22:16:37 <mtreinish> so instead of failing a run, it would look at the logs after jenkins reported the results and would leave another voting score (only +1 or -1) 22:16:51 <mtreinish> that way we wouldn't be hit by nondeterministic issues in the gate 22:17:00 <mtreinish> but we'd still get a -1 if there were errors in the logs 22:17:25 <dkranz> mtreinish: Will that cause any one to fix them? 22:17:59 <dkranz> mtreinish: A lot, if not most, of these errors are bugs. 22:18:07 <rahmu> mtreinish: will that be a blocking -1? 22:18:09 <mtreinish> dkranz: I don't know, we've never done something like that before 22:18:10 <dkranz> mtreinish: And they are easy to track down because you can see where they came from 22:18:26 <mtreinish> rahmu: no, it'd be a -1 like the third party testing 22:19:20 <clarkb> there is no such thing as a blocking -1 22:19:24 <clarkb> only -2 can block 22:19:34 <mtreinish> dkranz: yeah, it was just another approach to consider about doing this 22:19:39 <clarkb> (in the verified column) 22:19:54 <dkranz> mtreinish: Why can't we treat it like any other test failure, as we were doing? 22:20:07 <mtreinish> clarkb: isn't it any column? 22:20:25 <clarkb> mtreinish: well approved is only 0 or 1 so 0 is blocking 22:20:40 <mtreinish> clarkb: oh yeah that's true 22:21:04 <mtreinish> dkranz: it was more about splitting out the complexity from the one job I think 22:21:25 <dkranz> mtreinish: What complexity? I think we either care about this or we don't. 22:21:48 <dkranz> mtreinish: If I am the only one who really cares about it we should just drop the whole thing, no? 22:22:30 <mtreinish> dkranz: it's polluting the console log right now and everyone ignores it. So we have to climb the hurdle to get it failing jobs again 22:22:32 <dkranz> mtreinish: Having some bot is way more complex than what it was doing. 22:22:38 <mtreinish> this was just an idea for a middle ground 22:23:01 <dkranz> mtreinish: Only the oslo thing is polluting it. If we just fixed that there would not be a problem. 22:23:09 <dkranz> mtreinish: There wasn't a problem before. 22:23:28 <dkranz> mtreinish: ANd if we had left this on the oslo thing never would have gotten in! 22:24:00 <mtreinish> dkranz: I don't think that oslo.messaging is the only issue right now. It's the biggest one definitely 22:24:08 <mtreinish> and things slipped in because the script was broken for a while 22:24:15 <mtreinish> (on the d-g side) 22:24:22 <dkranz> mtreinish: Anyway, I am ok with moving on now 22:24:45 <mtreinish> dkranz: ok 22:24:50 <dkranz> mtreinish: If we are going to get anywhere, sdague will have to send an email about it. 22:25:09 <mtreinish> #topic Criteria for accepting tests that cannot run normally in the gate (dkranz) 22:25:16 <mtreinish> dkranz: this one is yours too 22:25:51 <dkranz> mtreinish: There could be a lot of valuable tests that we share but we can't due to our policy of only accepting code that runs upstream 22:26:04 <dkranz> mtreinish: I just thought we should clarify exactly what that means 22:26:22 <dkranz> mtreinish: So folks can decide whether to try to submit tests upstream or do them downstream, which would be a shame. 22:26:36 <mtreinish> dkranz: it's not code that runs upstream, we need results for every review with the test running 22:26:43 <mtreinish> it can be from an outside test system 22:26:52 <mtreinish> like the 3rd party testing requirements in the other projects 22:27:09 <mtreinish> the issue is that if we don't exercise tests for everything they tend to bitrot very quickly 22:27:13 <dkranz> mtreinish: So vote on every commit 22:27:26 <dkranz> mtreinish: That is a very high bar 22:27:30 <mtreinish> it's the same reason we stopped accepting commits with skips 22:27:37 <mtreinish> dkranz: otherwise we don't know if things work or not 22:27:43 <mtreinish> and that's not a good position to be 22:27:46 <dkranz> mtreinish: That may be ok for folks trying to get drivers into the code base 22:28:01 <mtreinish> right now we've got legacy tests in the tree like live migration that I've never run 22:28:07 <mtreinish> I have no idea if they work 22:28:19 <dkranz> mtreinish: I could not justify such a third-party system just to be able to submit my tests upstream 22:28:29 <dkranz> mtreinish: How about a compromise 22:28:42 <mtreinish> dkranz: it came up earlier this week with the multi-backend cinder 22:28:49 <dkranz> mtreinish: The tests can run be reported on by third party nightly 22:29:03 <dkranz> mtreinish: But if they stay broken for more than X time, they are removed 22:29:28 <mtreinish> dkranz: we tried that before with the nightly periodic all job 22:29:35 <mtreinish> no one ever looked at it 22:29:42 <dkranz> mtreinish: Not the remove part :) 22:30:01 <dkranz> mtreinish: That is the teeth 22:30:04 <mtreinish> dkranz: yeah after a few months I just ripped out all the tests that got run in all that didn't run in the gate 22:30:17 <mtreinish> I think it was mostly whitebox 22:30:19 <dkranz> mtreinish: RIght 22:30:37 <dkranz> mtreinish: And any one who cared enough to put the tests upstream would probably care enough to keep them working 22:30:48 <mtreinish> dkranz: it's a good idea but we'll have to be explicit about the policy 22:30:51 <dkranz> mtreinish: Just an idea 22:31:06 <mtreinish> and it'll require someone to watch it 22:31:19 <dkranz> mtreinish: ok, I'll send out some kind of proposal to the ml if it seems worthwhile 22:31:42 <mtreinish> dkranz: that hasn't been my experience. Things normally just get thrown over the fence 22:31:58 <mtreinish> dkranz: yeah bring this out to the ml 22:31:58 <dkranz> mtreinish: The test would be external so it is likely some one would be watching 22:32:08 <mtreinish> and maybe we'll follow up at summit 22:32:30 <dkranz> mtreinish: And one motivation for this is that our plan is to increase function of upstream 22:32:46 <dkranz> mtreinish: So in the future multnode tests maybe could run and if we do this they will be there 22:33:02 <dkranz> mtreinish: That's all for now 22:33:17 <mtreinish> dkranz: the issue with this though is the integrated gating, it's not just tempest that could break things 22:33:48 <mtreinish> dkranz: ok we can move on 22:34:01 <mtreinish> #topic Bugs 22:34:16 <mtreinish> Does anyone have any bugs that they think needs some attention? 22:34:30 <mtreinish> or any high priority or critical bugs that need extra eyes on them? 22:35:43 <mtreinish> ok I guess there aren't any bugs today :) 22:35:49 <afazekas> https://review.openstack.org/#/c/71575/ 22:35:55 <mtreinish> #topic Critical Reviews 22:36:01 <mtreinish> afazekas: good timing 22:36:11 <mtreinish> #link https://review.openstack.org/#/c/71575/ 22:36:23 <mtreinish> does anyone else have any reviews that they'd like to get some eyes on? 22:36:53 <dkranz> https://review.openstack.org/#/c/65930/ 22:37:09 <boris-42_> mtreinish #link https://review.openstack.org/#/c/70131/ 22:37:13 <dkranz> https://review.openstack.org/#/c/71579/ 22:37:14 <mtreinish> #link https://review.openstack.org/#/c/65930/ 22:37:36 <boris-42_> mtreinish it is not tempest but it is related .. 22:37:40 <mtreinish> #link https://review.openstack.org/#/c/71579/ 22:37:44 <mtreinish> boris-42_: that's fine 22:37:55 <mtreinish> I'll take a look at it probably tomorrow 22:38:12 <boris-42_> mtreinish could I share 2 blueprints around integration of rally & tempest? 22:38:46 <mtreinish> like share them between projects in lp? or right now in the meeting? 22:39:20 <boris-42_> mtreinish in meeting, could we have some topic about integration.. I would like to be a closer to openstack QA team.. 22:39:31 <boris-42_> mtreinish sorry didn't add it to agenda =( 22:39:39 <mtreinish> boris-42_: sure 22:39:49 <boris-42_> mtreinish thank you 22:39:49 <mtreinish> first does anyone else have reviews to bring up? 22:40:46 <mtreinish> ok I guess not 22:40:54 <mtreinish> #topic Rally tempest integration 22:40:59 <mtreinish> boris-42_: go ahead 22:41:13 <boris-42_> so there are 2 parts of integration 22:41:46 <boris-42_> first of all what is rally… small diagram https://wiki.openstack.org/w/images/e/ee/Rally-Actions.png 22:42:13 <boris-42_> so it is the tool that allows you to work with different clouds, verify them, deploy on (virtual) servers and benchmark 22:42:23 <boris-42_> (in future as well profile & analyze logs) 22:42:31 <boris-42_> it is done to simplify work for human 22:42:37 <boris-42_> =) 22:42:58 <boris-42_> We are trying to reuse as much as possible from OpenStack and related project 22:43:12 <boris-42_> e.g. one of deploy engine is based on DevStack 22:43:29 <boris-42_> so there are 2 first points that are related to tempest 22:43:41 <boris-42_> 1. add some kind of pretty interface to tempest 22:43:55 <boris-42_> https://blueprints.launchpad.net/rally/+spec/tempest-verification 22:44:07 <boris-42_> So when you are working around nova e.g. 22:44:10 <boris-42_> you have cloud 22:44:17 <boris-42_> you would like to have some command like 22:44:39 <boris-42_> rally verify nova (that will run only tempest tests that are related to nova) 22:44:51 <boris-42_> and after something fails 22:44:53 <boris-42_> you are fixing it 22:45:02 <boris-42_> and would like first of all to run failed tests 22:45:14 <boris-42_> so run rally verify latest_failed 22:45:31 <boris-42_> as well you would like to keep results for some cloud somewhere (it will be Rally DB) 22:45:42 <mtreinish> boris-42_: that sounds like a wrapper around a lot of things in tempest already 22:46:11 <mtreinish> boris-42_: We try to service tag tests so you can run with a regex filter compue for example and that should run every test that touches nova 22:46:30 <boris-42_> mtreinish yep buy I would like to simplify this step a bit 22:46:37 <boris-42_> mtreinish if it is already implemented in tempest great 22:46:41 <mtreinish> and testr already keeps a db (obviously a bit more simplistic than rally's) of runs with failed jobs 22:46:43 <boris-42_> mtreinish if it could be implement ok 22:46:46 <mtreinish> and information about them 22:47:26 <boris-42_> mtreinish I know but it is not enough simple for end users imho.. 22:47:49 <boris-42_> mtreinish there is a lot of tasty things that could be added 22:48:07 <boris-42_> mtreinish one more time the goal is not to reimplement stuff (just unify & simplify interface) 22:48:20 <boris-42_> mtreinish and somewhere store all results related to specifc cloud 22:48:35 <mtreinish> boris-42_: ok, I just don't know if those 2 examples have to be rally specific 22:48:46 <mtreinish> they seem generally applicable to tempest and testr and improvements 22:49:13 <boris-42_> mtreinish we will try to implement all that is possible inside tempest 22:49:15 <mtreinish> I don't know if we should wrap things to add extra functionality 22:49:21 <mtreinish> boris-42_: ok 22:49:29 <boris-42_> mtreinish the idea is next 22:49:53 <boris-42_> mtreinish it is nice when you have one interface for all operation that is all unified 22:50:17 <boris-42_> mtreinish I mean one commad to add/deploy cloud 22:50:24 <boris-42_> mtreinish then another command to play with tempest 22:50:32 <boris-42_> mtreinish then third command to benchmark 22:50:42 <boris-42_> mtreinish and even after year you will have all results 22:50:54 <boris-42_> mtreinish in one place 22:51:48 <boris-42_> So goal is to unify, make some pretty hooks for most often commands, somewhere store results and so on=) 22:52:37 <boris-42_> mtreinish as well as automation of generation of config for tempest by passing endpoints of cloud 22:53:24 <boris-42_> I know when you are working with tempest for a while it is just a put here and there some info and it works 22:53:47 <boris-42_> but when somebody is newbie to tempest it takes a while.. 22:54:10 <boris-42_> So next thing is https://blueprints.launchpad.net/rally/+spec/tempest-benchmark-scenario 22:54:10 <mtreinish> boris-42_: we've actually been working on tooling to simplify that part of the problem 22:54:27 <boris-42_> mtreinish oh it will be nice if you share your results/blueprints/discussion 22:54:39 <boris-42_> mtreinish we will be glad to help you guys 22:55:06 <mtreinish> boris-42_: this is most recent one I'm working on: https://blueprints.launchpad.net/tempest/+spec/config-verification 22:55:32 <mtreinish> boris-42_: you definitely have a lot of info to share here, but we're running out of time 22:55:38 * boris-42_ mtreinish added to bookmarks 22:55:47 <mtreinish> and it feels like we need a larger discussion about rally and tempest 22:55:47 <boris-42_> so okay just a bit about benchmarking scenario 22:55:52 <rahmu> mtreinish: speaking of which. Can you give us a quick status about it and tell us what's left to do? 22:56:04 <rahmu> mtreinish: I'm talking about the config-verification bp 22:56:10 <mtreinish> boris-42_: can you take this to the ML 22:56:10 <boris-42_> mtreinish okay we can continue in mailing list probably?) 22:56:16 <boris-42_> mtreinish sure I will take 22:56:16 <mtreinish> with all the details 22:56:24 <mtreinish> boris-42_: great thanks 22:56:24 <boris-42_> mtreinish let us then prepare demo 22:56:39 <boris-42_> mtreinish we are always able just to put some code from rally to tempst/devstack 22:56:51 <boris-42_> actually we will be only glad to reduce code base of rally=) 22:57:24 <mtreinish> rahmu: sure I'm still working on adding all the extension detection. I've got a patch up for swift now 22:57:40 <mtreinish> then I still need to finish api version discovery for keystone, and cinder 22:57:57 <mtreinish> and add endpoint/service checking to it as well 22:58:08 <mtreinish> and I'm sure there are other optional features or things I'm missing 22:58:17 <mtreinish> but that's what I have on my mind right now for it 22:58:21 <mtreinish> boris-42_: ok cool 22:58:34 <boris-42_> mtreinish thank you for giving timeframe=) 22:58:54 <mtreinish> #topic open discussion 22:59:07 <mtreinish> actually there is only ~1 min left 22:59:11 <mtreinish> so let's end here today 22:59:13 <boris-42_> ^_^ 22:59:20 <dkranz> bye 22:59:23 <mtreinish> if there was something we couldn't get to we can just pick it up on -qa 22:59:26 <rahmu> bye everyone 22:59:27 <mtreinish> thanks everyone 22:59:29 <mtreinish> #endmeeting