17:00:06 <mtreinish> #startmeeting qa 17:00:07 <openstack> Meeting started Thu Jan 30 17:00:06 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:11 <openstack> The meeting name has been set to 'qa' 17:00:18 <mtreinish> hi who's here today? 17:00:21 <sdague> o/ 17:00:50 <mtreinish> heh, ok this'll be a quick one I guess 17:01:00 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 17:01:07 <mtreinish> ^^^ today's agenda 17:01:15 <dkranz> here 17:01:15 <andreaf> hi 17:01:19 <rahmu> hi 17:01:37 <mtreinish> so it looks like today's agenda is just the boiler plate one 17:01:41 <mtreinish> but let's dive into it 17:01:47 <mtreinish> #topic Blueprints 17:02:04 <mtreinish> so does anyone have any blueprints to bring up that need attention 17:02:11 <mtreinish> or that have a status update 17:02:26 <andreaf> ok I'll start here 17:02:40 <andreaf> I have an implementation of the multi auth bp here https://review.openstack.org/#/c/65311/ 17:02:43 <giulivo> hi 17:03:34 <andreaf> I think it would be good to get that through in icehouse as keystone v2 is going to be deprecated 17:04:04 <mtreinish> andreaf: can your massive refactor wait until after mine? :) https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/config-cleanup,n,z 17:04:09 <mtreinish> it's clash of the titans 17:04:23 <dolphm> o/ identity api v2 has been officially deprecated as of icehouse-m1 17:04:29 <andreaf> it's quite a large patch, a pain to rebase, so if anyone has ideas on how to split it ... 17:04:41 <andreaf> mtreinish: yes I think so 17:04:48 <sdague> dolphm: what is the status of the official python clients supporting v3? 17:05:29 <andreaf> mtreinish: I saw your patch both together will make the rest client __ini__ down to one parameter :) 17:05:45 <dolphm> sdague: python-keystoneclient supports it quite well from a library perspective; python-openstackclient exposes it quite well on the CLI, barring some initial authentication/configuration hurdles which represents a significant barrier to adoption 17:06:03 <sdague> dolphm: so if you use python-novaclient 17:06:09 <sdague> does it call keystone v3? 17:06:20 <dolphm> not today 17:06:23 <andreaf> sdague: I fear not 17:06:29 <sdague> what's required to make that happen? 17:06:40 <andreaf> most clients allow token / base url as input 17:06:44 <andreaf> but not nova / cinder 17:06:50 <andreaf> I filed a bug for that 17:06:59 <sdague> because I'm a little concerned about use wildly diverging here from the way people are going to use this in the field 17:07:08 <andreaf> I think all clients shall rely on keystone client for auth 17:07:10 <dkranz> This is really going to put the "client libraries should work with old openstack releases" to the test... 17:07:26 <mtreinish> sdague: well for the api tests at least there shouldn't be an issue 17:07:42 <sdague> mtreinish: sure, as long as we decide to run the v2 stuff still 17:07:50 <sdague> both ways that is 17:08:08 <andreaf> https://bugs.launchpad.net/python-novaclient/+bug/1262843 17:08:09 <mtreinish> sdague: yeah I thought that was the intent having one job with v2 and the other with v3 17:08:19 <dolphm> andreaf: we're working on exposing some neat library features (mostly taking advantage of requests "sessions") to make it much easier for other clients to fully adopt identity api v3 in a couple lines of code 17:08:19 <mtreinish> at least while v2 is still in tree 17:08:23 <sdague> right, we do start to run up against job limits 17:08:33 <sdague> so I'm cool with the cleanups this provides 17:08:41 <sdague> and the plugability 17:08:47 <dkranz> sdague: +1 17:08:49 <sdague> but I don't actually think this is icehouse critical 17:09:04 <sdague> so if it doesn't all make it, that's fine. 17:09:18 <mtreinish> sdague: I bumped it up from low to high(not critical) yesterday 17:09:33 <mkoderer> hi, sry I am late 17:09:59 <sdague> because there is clearly a missing piece in all this with the fact that we can't actually run a v3 job 17:10:05 <sdague> because we'll be using v2 all over the place 17:10:20 <sdague> or v3 only job 17:11:07 <sdague> so lets get as much as makes sense in, but this really seems to beg the question of the overall matrix, which I think is going to be a summit discussion 17:11:48 <mtreinish> yeah we need to iron out the details on the matrix 17:12:04 <mlavalle> Hi, sorry for the delay 17:12:10 <andreaf> sdague: r3 v only job I agree, but at least we'll be stressing v3 a bit more 17:12:16 <sdague> andreaf: sure 17:12:37 <mtreinish> andreaf: ok is there anything else on the keystone v3 blueprint? 17:12:53 <andreaf> andreaf: no 17:13:00 <andreaf> mtreinish: no 17:13:07 <mtreinish> heh, ok 17:13:10 <andreaf> can't make sense in my typing :P 17:13:19 <mtreinish> does anyone else have a blueprint to bring up 17:13:19 <mtreinish> ? 17:13:24 <dkranz> mtreinish: negative test 17:13:28 <mkoderer> yep 17:13:36 <dkranz> mtreinish: It is waiting for core reviews 17:13:36 <mtreinish> dkranz, mkoderer: ok go ahead 17:13:42 <mkoderer> please review https://review.openstack.org/#/c/64733/ 17:13:59 <mkoderer> I think we should merge it and fix the nits later 17:14:12 <sdague> I'm ok with that 17:14:27 <sdague> let's give people till end of day to raise objections 17:14:30 <dkranz> So it has three +1 and just needs some +2 17:14:38 <sdague> I'll take a look this afternoon 17:14:45 <mkoderer> sdague: great 17:14:46 <sdague> if there are none, I say we merge it 17:14:47 <dkranz> sdague: Thanks 17:14:52 <sdague> and fix the rest in tree 17:14:54 <mtreinish> mkoderer: yeah I've been meaning to take a closer look at it. The only concern I had is with the jsonschema being in .py files 17:15:00 <mtreinish> why not just make it a .json file? 17:15:22 <dkranz> mtreinish: The problem is that the tests have to reference a schema by name 17:15:22 <sdague> yeh, that is the one cleanup I'd kind of like first 17:15:23 <mkoderer> that was dkranz idea ;) 17:15:37 <dkranz> sdague: The json was moved to separate files 17:15:40 <sdague> dkranz: use descriptive file names 17:16:05 <dkranz> sdague: You mean a separate file for each schema? 17:16:08 <sdague> yeh, I'd still rather they be real json. 17:16:11 <sdague> dkranz: sure 17:16:14 <sdague> whatever makes sense 17:16:25 <dkranz> sdague: I didn't realize you were thinking of hundreds of new files for thies 17:17:05 <sdague> Also if they are real json, they are much more easily syncable from nova 17:17:27 <mtreinish> dkranz: yeah if we eventually have an automated method for generating them then it isn't a big deal 17:17:32 <mkoderer> sdague: ok well then let's move it 17:17:39 <sdague> oh, actually they aren't 17:17:54 <sdague> well, I think we should get to real json as separate files anyway :) 17:18:05 <sdague> so if you guys make that change, I'm +2, and we do the rest in tree 17:18:06 <dkranz> sdague: I don't object, just want to make sure we accept having a separate file for each item at api.openstack.org 17:18:18 <sdague> dkranz: yep 17:18:28 <dkranz> sdague: ok, I'm in 17:18:41 <rahmu> and are each of these files written manually or autogenerated? 17:18:54 <mkoderer> ok, I think I will fix the little nits right away and push it tomorrow 17:19:14 <dkranz> rahmu: Manually, at least for now. From what would you generate them? 17:19:14 <mtreinish> rahmu: right now they're manual but eventually there will be a script to manually create them 17:19:46 <dkranz> mtreinish: I still don't understand from what you could generate them 17:19:54 <rahmu> okay, I saw mention of syncing from nova, and didn't understand 17:20:01 <dkranz> mtreinish: They are the source from which other interesting things could be generated 17:20:17 <dkranz> mtreinish: But it would be better if they were defined in projects rather than tempest 17:20:21 <mkoderer> I don't think that you can generate them completely 17:20:27 <sdague> dkranz: you should be able to reuse a big part of the validation schema from nova, no? 17:20:34 <sdague> for the pieces that are done there 17:21:06 <dkranz> sdague: maybe, but I am not convinced. I would say it should really be the other way around 17:21:09 <sdague> I realize that today the efforts are pretty discreet 17:21:13 <mtreinish> dkranz: if the schema is queriable then all the details can be queried from nova 17:21:36 <sdague> dkranz: right the only tempest specific pieces would be the logic to go after interesting boundaries, right? 17:21:37 <mkoderer> mtreinish: I don't think that you get all the needed details 17:21:38 <dkranz> mtreinish: Our schema has more information, containing what is in nova as a subset 17:22:01 <sdague> dkranz: ok, so we definitely need to figure out how to converge those more, but we can do that in tree 17:22:08 <sdague> I accept this is a good starting point 17:22:10 <dkranz> mtreinish: The nova methodology is flawed because it did non consider all the use cases for schema 17:22:11 <mkoderer> such as expected result code for malicious input 17:22:33 <mtreinish> dkranz: I'm looking at it, you have: name, method, url, and the json schema in the files 17:22:34 <sdague> dkranz: ok, so lets figure out how to get that handled on the nova side as well 17:22:47 <dkranz> sdague: great 17:22:56 <mkoderer> ok 17:22:57 <mtreinish> if the schema is queriable what can't be done with a script? 17:22:57 <dkranz> sdague: Of course the other zillion apis have nothing at all 17:22:59 <sdague> because nova will need to be the authority on their API 17:23:05 <sdague> dkranz: agreed 17:23:16 <sdague> but at least with nova we should try to find a way to leverage the work over there 17:23:24 <sdague> then hopefully other interfaces will follow 17:23:31 <dkranz> mtreinish: It is not script or no script, just that the nova schema just describes payload 17:23:52 <mkoderer> mtreinish: it doesn't care about result codes 17:23:53 <dkranz> mtreinish: It's not enough to generate tests 17:23:54 <sdague> #action mkoderer to break out json schemas into .json files 17:24:12 <mkoderer> sdague: ok cool 17:24:15 <dkranz> mtreinish: They simply did not consider the more general use cases for schemas 17:24:35 <sdague> ok, I think we can move on :) lets get the change, get this landed, realize we're going to need summit time to work out some of the details on schema reuse 17:24:45 <dkranz> sdague: yes 17:24:51 <mkoderer> sdague: +1 17:24:54 <mtreinish> ok yeah that's fair 17:25:08 <mtreinish> oh I see now it's not just the jsonschema in the dict you're adding to it 17:25:10 <sdague> but it's definitely moving us in the right direction 17:25:15 <sdague> progress 17:25:27 <sdague> similar thing.... 17:25:30 <dkranz> sdague: The nova schema is not in files as real json :) 17:25:32 <mkoderer> mtreinish: exactly.. 17:25:37 <sdague> mtreinish's config patch 17:25:45 <giulivo> sorry guys a question from an outsider, when you say that nova will be an authority over the other apis, it means we expect to continue to have facilities in the nova api to create resources of other type (like networks or images), and not to strip it from nova, is that correct_ 17:25:51 <mtreinish> ok well I've got a quick status update on the config bp 17:25:53 <sdague> please take a look, because it does a ton of clean up 17:25:57 <mtreinish> oh sdague beat me to it 17:26:13 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/config-cleanup,n,z 17:26:27 <sdague> and I'd like some eyes on it to make sure it's fine, then probably squash and bulk merge it 17:26:32 <mtreinish> I have one more patch on top to add 17:26:35 <dkranz> giulivo: NO, it has nothing to do with that. 17:26:55 <sdague> mtreinish: is that the patch that finally removes all the self.config assignments? 17:26:57 <mtreinish> I'm holding off because of rebase complications because masayukig approved some 17:26:58 <giulivo> ok so I must have missed something, sorry 17:27:02 <sdague> I hadn't see them yet 17:27:03 <dkranz> giulivo: Just that the api schema should be defined in a project, not in tempest ideally 17:27:04 <mtreinish> sdague: yeah 17:27:19 <mtreinish> it removes the references from manager.py and test.py 17:27:38 <sdague> honestly, unless there is an objection, I'm just going to +A that through once it looks like it works 17:27:51 <sdague> because it's going to be a rebase monstrosity 17:28:24 <sdague> and I'd rather spend our time on useful things, not rebase chasing that series 17:28:26 <mtreinish> sdague: yeah well if I push things now the sha1's on the ones in the queue will change because there is already some of that 17:28:58 <mtreinish> I was trying to make things manageable I hate giant patches 17:29:23 <mtreinish> ok I don't have anything else on this 17:29:35 <mtreinish> let move to the next topic 17:29:35 <sdague> mtreinish: get a working top patch, then we can figure out how to best get this in 17:29:44 <mtreinish> sdague: ok 17:29:52 <mtreinish> #topic Neutron testing 17:30:02 <mlavalle> Hi 17:30:03 <mtreinish> mlavalle, salv-orlando: you around? 17:30:18 <mtreinish> are there any updates on neutron testing? 17:30:20 <mlavalle> api tests development is going well 17:30:40 <mlavalle> we now have a community of debs actively engaged 17:30:50 <mlavalle> devs ^^^ 17:31:06 <mlavalle> and I have been doing a lot of code reviews 17:31:09 <sdague> mlavalle: nice job 17:31:31 <sdague> is there any understanding on why the isolated jobs remain at a really high failure rate? 17:31:50 <salv-orlando> mtreinish: the patches are still waiting for the neutron gate to get back to a point where we can merge anything 17:31:58 <mlavalle> I haven't worked on that, but salv-orlando might have insight 17:32:01 <mtreinish> sdague: I thought that was the kernel namespace bug 17:32:10 <mtreinish> salv-orlando: which patches? 17:32:26 <salv-orlando> the patches for parallel testing 17:32:40 <salv-orlando> and for enabling full isolation 17:33:02 <salv-orlando> but since we are stuck on this kernel bug we can't make any progress until that is solved. 17:33:16 <sdague> salv-orlando: does that really account for all the fails? 17:33:30 <sdague> I know that's an impact 17:33:43 <sdague> but I was curious if it was the complete problem, or just one of a couple 17:34:10 <salv-orlando> sdague: not all the fails. Bug 1253896 astir lurks. But bug 1273386 is a lot more frequent 17:34:12 <sdague> also, any word on the new kernel ? 17:34:27 <sdague> and if that looks stable enough to use more broadly 17:34:33 <salv-orlando> sdague: we see the failure that too. And that's bad. 17:34:40 <sdague> ok 17:34:52 <sdague> yep, so at least we know that's not the answer 17:34:57 <mtreinish> sdague: I saw patches, from russellb I think, adding it to nodepool 17:35:08 <salv-orlando> mtreinish: experimental jobs now run on 3.11 17:35:21 <sdague> right, I wanted to know if we had enough data to sort it if it fixed 17:35:24 <mtreinish> salv-orlando: ok cool 17:35:27 <sdague> and it sounds like we do not 17:35:41 <sdague> or we do have enough data.... and it did not 17:36:03 <salv-orlando> yes sdague: we hit the bug at the 3rd recheck attempt 17:36:10 <sdague> ok 17:36:13 <afazekas> BTE anybody else see similar issues: https://bugs.launchpad.net/neutron/+bug/1274121 17:36:51 <russellb> salv-orlando: any need to keep the test nodes with 3.11? 17:36:52 <giulivo> I wonder if there are plans to upgrade later this year to the new LTS, current ubuntu is preventing other stuff from being tested too 17:37:07 <salv-orlando> afazekas: first time i see that failure 17:37:20 <sdague> giulivo: so I'm sure the 14.04 test story will be a Juno topic 17:38:00 <sdague> ok, probably worth moving on 17:38:03 <salv-orlando> russellb: it might be worth keeping them for a while, but since the stack trace is exactly the same I guess there's little we can do. 17:38:17 <salv-orlando> Actually.. I would try to push back 3.2.0-57 in experimental nodepool 17:38:26 <sdague> salv-orlando: that's a good idea 17:39:08 <sdague> because the other possibility is this existed a lot longer than we realized, and we just didn't have the tooling to see it 17:39:09 <giulivo> sdague, I'll try harder to join this time 17:39:15 <salv-orlando> because internally I've been unable to reproduce either on 3.2.0-57 and 3.2.0-58 but going back in time the first time we saw this issue it was with kernel-3.2.0-54 17:40:00 <sdague> yeh, I expect this may be a case of a problem that we didn't know we had until recently, and would explain overall neutron gate stability 17:40:10 <sdague> giulivo: cool 17:40:25 <sdague> mtreinish: ok, what's next? :) 17:40:43 <sdague> actually, before that, who's volunteering to handle moving the experimental kernel? 17:40:43 <mtreinish> #topic Bugs 17:40:58 <sdague> let's make sure we get a volunteer on that 17:40:59 <mtreinish> oh what's the fancy undo command? 17:41:50 <sdague> #info need volunteer to put experimental nodepool on 3.2.0-57 kernel 17:41:56 <sdague> ok, I guess we move on 17:42:02 <mtreinish> ok 17:42:15 <mtreinish> so does anyone have any bugs that need attention 17:42:20 <NithyaG__> Hi, I have a query on Nova tests. I would like to know if its a good idea to get a few additional tests on live migration added, though its only single node deployment in gate as of today, it will be good to build the test repository. I have raised a few bugs today for adding test cases. (both API and scenario tests) 17:42:24 <mtreinish> or are there any critical bugs not getting any attention 17:42:55 <sdague> NithyaG__: so the problem is we don't really have any way to validate them 17:44:02 <sdague> guess no one needs bug attention 17:44:05 <sdague> moving on :) 17:44:05 <andreaf> sdague: how's the work on multinode gate proceeding? 17:44:09 <mtreinish> ok then let's move on 17:44:15 <sdague> anderstj: no one is working on it 17:44:28 <sdague> andreaf: that should have been 17:44:37 <andreaf> sdague: oh I see 17:44:39 <dkranz> sdague: Is there a description of the issues with multinode anywhere? 17:44:50 <dkranz> sdague: Or some unimplemented blueprint or plan? 17:44:57 <andreaf> andreaf: we already have a few live migration and resize tests in 17:45:05 <sdague> dkranz: not that I know of, no one has ever taken that blueprint as their primary task 17:45:06 <andreaf> sdague: I meant 17:45:20 <sdague> andreaf: so resize we can do in the gate 17:45:23 <dkranz> sdague: ok, I get it 17:45:26 <sdague> because you can resize to yourself 17:45:42 <sdague> we added a flag to nova to let that happen in the gate 17:45:59 <mtreinish> andreaf: I've never run live migration and I've never heard of anyone using it 17:46:06 <mtreinish> I'm sure the code has a lot of bit rot 17:46:11 <mtreinish> since it doesn't get run in the gate 17:46:14 <andreaf> sdague: oh, ok 17:46:35 <mtreinish> ok we can save this for later if there is anything else on it 17:46:39 <sdague> yep 17:46:40 <mtreinish> #topic Critical Reviews 17:46:52 <mtreinish> does anyone have any reviews that they would like to bring up? 17:47:24 <sdague> only the ones we talked about in blueprints 17:47:35 <mtreinish> yeah I wasn't going to bring those up again 17:48:01 <mtreinish> I guess there aren't any others then 17:48:03 <mlavalle> mtreinish: over the weekend I will put together a list of Neutron api tests that need core reviewers a ttention. I will send it to the ML 17:48:19 <sdague> sounds great 17:48:34 <afazekas> may be I missed the blueprint parts, but this https://review.openstack.org/#/c/65311/ definitely should get more ayes 17:48:36 <mtreinish> mlavalle: ok sounds good. But I'm hesitant to merge anything until the neutron gate is in a better state though 17:48:37 <sdague> also, given the long run times due to the concurency drop 17:48:53 <sdague> please keep an eye on new tests that take a long time, we shouldn't be adding them right now 17:49:14 <rahmu> it would be great to get some input on this one: https://review.openstack.org/#/c/62887/ We'd like to know how to avoid hardcoding the projects' names 17:49:20 <sdague> hopefully some nova fixes will go in and let us run faster again, but until then, lets not kill ourselves too hard 17:49:23 <dkranz> sdague: So what about a test that takes 5s? 17:49:24 <mtreinish> #action sdague to send an email to list about test run time and reviewing 17:49:28 <andreaf> afazekas: thanks yes I mentioned that during the bp part 17:49:32 <mlavalle> ok, I'll be careful with that. I'll focus on api tests that don't take long 17:49:33 <sdague> dkranz: 5s isn't troubly to me 17:49:44 <sdague> 30s is 17:49:55 <dkranz> sdague: ok, reasonable 17:50:20 <sdague> afazekas: we talked about 65311 previously 17:50:27 <mtreinish> #link https://review.openstack.org/#/c/62887/ 17:50:30 <dkranz> sdague: I am actually suspicious that our setup time is getting large and could be improved 17:50:37 <mtreinish> rahmu: I wonder if that should really be a bp instead 17:50:56 <dkranz> sdague: The sequential output of tests that do almost nothing is quite slow 17:50:58 <rahmu> mtreinish: that's the kind of attention this patch also needs :) 17:51:03 <giulivo> well I've one which surely is going to take time but it's the first and only test for this feature I think: https://review.openstack.org/#/c/67616/ 17:51:10 <dkranz> sdague: I am going to look at that 17:51:34 <sdague> dkranz: so I'm actually working on something with lifeless's help that I think makes it a little easier to see what's going on 17:51:45 <sdague> https://review.openstack.org/#/c/70111/ 17:51:51 <mtreinish> giulivo: yeah that will add some run time probably :) 17:52:03 <sdague> oh, on that front, the basic ops test 17:52:07 <sdague> that we run on 3 images 17:52:14 <sdague> why do we do that again? 17:52:20 <giulivo> mtreinish, I can remove the gate tag but we didn't have anything about backups so ... 17:52:38 <mtreinish> giulivo: removing the gate tag won't do anything actually 17:52:57 <dkranz> sdague: I will poke around some on these issues. It has been a while since we looked at such things 17:52:58 <giulivo> sorry yeah , add slow 17:53:30 <mtreinish> sdague: it was a feature add recently for running with different image types which in the gate really isn't that useful 17:53:42 <sdague> right, I see the poc value 17:53:49 <sdague> but in the gate, I think we should turn that off 17:53:55 <mtreinish> I think andreaf made it configurable 17:54:01 <sdague> it's not really catching anything, and just takes up time 17:54:08 <mtreinish> so we should just use it for one image type 17:54:13 <sdague> I was also wondering about the loader 17:54:19 <sdague> so where we skip tests 17:54:31 <sdague> skip whole classes 17:54:35 <andreaf> mtreinsh, sdague: yes it is configurable which images and which flavours 17:54:44 <sdague> is it possible to make the load just not register those classes? 17:55:05 <sdague> andreaf: ok, could you propose something which defaults it back to only 1? 17:55:17 <andreaf> sdague: ok will do 17:55:21 <sdague> thank you 17:55:36 <sdague> mtreinish: thoughts on doing class level skip by the loader? 17:55:57 <mtreinish> sdague: I don't think that's how skip works. I think it just catches the exception and goes from there. I have to double check 17:56:07 <sdague> mtreinish: I know that's not how skip works 17:56:21 <mtreinish> sdague: what exactly are you proposing? 17:57:10 <mtreinish> sdague: how about we save the discussion for after the meeting 17:57:25 <mtreinish> because we're down to ~3 min? 17:57:26 <sdague> is it possible to have test_discover not discover a set of classes if we decorate them correctly 17:57:28 <sdague> sure 17:58:02 <mtreinish> sdague: oh I'm not sure on that, maybe 17:58:18 <sdague> it would give us less chance of completely unbalancing testr 17:58:28 <giulivo> I have this crazy idea that we should slowly drop from nova api what should be done using $service api and as a consequence, remove duplicated tests 17:58:30 <mtreinish> we can set patterns to match 17:58:38 <mtreinish> so maybe we can play games there 17:58:43 <sdague> yep 17:58:49 <sdague> ok, worth exploring 17:59:06 <sdague> anything else from folks? 17:59:52 <mtreinish> ok I guess we'll end here since we're out of time 17:59:56 <mtreinish> thanks everyone 17:59:57 <sdague> ack 17:59:59 <sdague> thanks everyone 17:59:59 <mtreinish> #endmeeting