22:00:18 #startmeeting qa 22:00:19 Meeting started Thu Jan 23 22:00:18 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:22 The meeting name has been set to 'qa' 22:00:40 hi, who's here? 22:00:43 hi 22:00:49 hi 22:00:49 hi 22:00:50 o/ 22:00:50 o/ 22:00:53 o/ 22:01:11 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 22:01:18 ^^^ today's agenda 22:01:27 hey 22:01:34 hey 22:01:47 so let's get started 22:01:51 #topic Gate Updates (sdague) 22:01:58 sdague: you're up 22:02:01 great 22:02:12 so I wanted to make sure people saw the email on this one 22:02:19 Yup 22:02:20 hi 22:02:26 New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html 22:02:34 #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html 22:02:35 #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html 22:02:37 heh 22:02:40 race condition! 22:02:42 I win... 22:03:01 basically, as people have seen, we remain extremely backed up in the gate 22:03:10 and it's a situation that we don't expect to get better 22:03:17 in terms of overall load 22:03:25 we can tackle some of the bugs, and that will help 22:03:36 but the load in terms of patches keeps going up at every milestone 22:03:57 and will continue to go up even more 22:04:04 so we've got some relatively extensive changes proposed to make i3 viable 22:04:33 mostly, I wanted to point out the email, and open up for comments discussion 22:05:08 sdague: it's sounds good to me 22:05:25 Yes. I was afraid we were on a death march 22:05:26 I'm curious to see how this shifts pain points though 22:05:35 agreed, it is an experiment 22:05:40 and I expect we'll adjust over time 22:06:06 I do think that at some point we will need to go to a per-project branch/merge model like the linux kernel 22:06:09 the one thing we've learned, is we hit new scale levels on every cycle, so have to rethink things from time to time 22:06:12 But this is a good start 22:06:26 dkranz: we do need some way to test openstack as a whole 22:06:34 because when we haven't, it doesn't work 22:06:52 sdague: Yes, I had an alternate proposal but I am not going to propose it now because progress is being made 22:06:52 Run a slimmer set of jobs in the gate queue to - means reduce gated job? 22:06:53 like when ceilometer turned out it was crashing nova if installed 22:07:22 ravikumar_hp: my expectation is we'll at least run tempest-full and grenade 22:07:31 but not in a bunch of different configs 22:07:41 and also, not all the project specific tests, like unit tests 22:08:03 It is worth trying 22:08:16 "Make a clean recent Check prereq for entering gate" How do you define "recent" and who will trigger the check? 22:08:17 the clean check scorecard, and auto recheck should keep things just about as functional 22:08:27 rahmu: recent == 24hrs 22:08:37 and zuul will trigger it itself if it's out of date 22:08:47 so a +A with old test results will run another check 22:08:53 and if it passes, move to gate 22:08:56 automatically 22:09:04 sdague: aren't you afraid that this might increase the load on the check gate drastically? 22:09:14 check queue* 22:09:27 rahmu: it will, however load in the check queue is less problematic 22:09:32 It is not just load 22:09:33 because the jobs are independent 22:09:53 the big reset costs of the speculation queue are what's weighing us down now 22:10:34 this will also help solve a currently unsolved problem, which is reviewers pushing +A before they have test results, or with month old test results 22:10:42 which is actually happening quite a bit 22:10:53 and leading to gate instability 22:11:21 sdague: Good. It is annoying to look at a review you want to +A but you have to wait with no efficient way to be notified when it is ok. 22:11:23 I think I saw one change in the gate with the last jenkins run from Oct 13 on monday 22:11:38 dkranz: right, this will mean you don't need to 22:12:01 +A, it will do all the right things 22:12:31 sdague: Oct. 13, yeah I'm sure that won't have a merge conflict :) 22:12:42 it actually didn't 22:12:47 :) 22:12:49 it was a stable/havana glance change 22:13:06 however, it couldn't pass because stable/grizzly devstack was not good 22:13:39 heh, ok 22:13:47 we'll probably also put some resweep logic in there so that code that's getting reviewed if it hasn't seen results in the past week will rerun as well 22:14:03 so test results will always be reasonably up to date 22:14:15 sdague: is that where all of mikal recheck no bug comments are coming from? 22:14:17 but that's a later optimization 22:14:22 mtreinish: so he did it out of band 22:14:28 this will be a feature of zuul 22:14:31 sdague: When are these changes taking effect? 22:14:35 sdague: oh, ok that makes sense 22:14:43 as soon as jeblair gets the zuul bits solid 22:14:48 possibly as early as this weekend 22:14:55 if not, within a couple weeks 22:14:55 sdague: cool 22:15:16 in time to fix issues on the system prior to i3 22:15:37 there have been other things going on as well, zuul now does a slow start on the dependent queue 22:15:45 so our reset cost is less 22:16:01 only the top N changes will get run, and that will get dynamically adjusted as we pass or fail 22:16:06 (zuul changes are ready to go) 22:16:11 jeblair: awesome 22:16:30 jeblair: nice 22:16:47 #link https://review.openstack.org/#/c/68516/ 22:16:57 ok, that's probably enough on gate, unless there are other questions / comments 22:17:26 ok let's move on then 22:17:33 #topic Blueprints 22:17:50 does anyone have any status updates on blueprints that they'd like to bring up 22:18:07 just the one I highlighted in the agenda 22:18:11 dkranz: I think you put an entry for the negative test stuff 22:18:13 dkranz: yep 22:18:18 let me throw one other one in 22:18:35 mtreinish: Are we still using blueprints for test development tasks? 22:18:47 which is actually a nova blueprint - https://blueprints.launchpad.net/nova/+spec/remove-v3-xml-api 22:19:00 ravikumar_hp: I think we said one per project 22:19:09 ravikumar_hp: yes for certain classes of tasks 22:19:11 as a master blueprint 22:19:12 but I've tagged a couple of tempest changes to it, as we'll need to move those through as part of it 22:19:28 https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/remove-v3-xml-api,n,z 22:19:45 sdague: ok we can touch that after dkranz's bp 22:19:47 sure 22:19:59 #topic negative tests bp 22:20:03 dkranz: ok go ahead 22:20:14 mtreinish: It is related in that xml was the reason for the current -1 on the negative test patch 22:20:28 So I put four questions in the agenda 22:20:54 Marc and I tried to push something with enough meat that folks could really see the whole picture 22:21:14 ping devananda 22:21:24 dkranz: ok, yeah the xml discussion is only for v3 22:21:28 max_lobur: pong 22:21:37 you'll still need it for the v2 side of the negative testing 22:21:38 Yes, but it is the same issue with negative 22:21:55 dkranz: what's the crux of the -1? 22:21:56 but since there isn't a schema available for v2 I guess it's not really a discussion 22:21:57 Do we want to maintain xml generation and schemas for the negative tests? 22:22:12 dkranz: in my opinion, no 22:22:24 I think we drop the xml negative tests completely 22:22:36 sdague: we probably should keep it for v2 22:22:41 as long as we keep v2 around 22:22:46 in tempest 22:22:50 mtreinish: for now, sure, I don't want to increase it though 22:22:57 sdague: yeah definitely 22:23:20 I'm ok with keeping the negative xml tests that exist for apis 22:23:26 dkranz: this was actually part and parcel for why I've been pushing hard to get a TC level recommendation to just do JSON 22:23:31 I just don't want to add xml auto-generation 22:23:40 dkranz: yes, agreed 22:23:44 sdague: I agree with you 22:24:05 dkranz: for the question in the agenda on schema location I'd say an external file would be best 22:24:05 sdague: So I think the rest of the -1 were little things that we will fix 22:24:13 IMO it just clutters up the test files 22:24:23 mtreinish: external to tempest? 22:24:35 no in tree but outside the test file 22:24:42 like in etc or something 22:24:44 mtreinish: that sounds good 22:24:53 or even in the tree in a well known location 22:25:00 etc is something people might think they can change 22:25:02 mtreinish: I think it would be good to keep them in the api tree with their pservices 22:25:03 dkranz: also that helps if we add an autogeneration tool to refresh the schemas 22:25:15 but in separate files 22:25:21 sdague: yeah that's a good point 22:25:21 dkranz: yes, I think I agree 22:25:41 dkranz: +1 22:25:54 Ideally these schemas would have other uses 22:26:10 like generating api doc 22:26:14 yeh, I think eventually we'll get to a place where we can pull the schemas from the service 22:26:24 Really what I would like to see is these be required as part of new apis 22:26:28 but that's a ways down the road 22:26:33 ken1ohmichi: are you working on making the schema queriable in nova? 22:26:42 I think that doing this in nova will mean it spreads 22:26:48 yes, a lot of patches for it. 22:27:02 nova tends to set the tone for a lot of things 22:27:02 ken1ohmichi: ok cool, I just wanted to check 22:27:10 Remember that those schemas are only for payload and not enough for testgen 22:27:19 dkranz: right 22:27:23 true 22:27:24 The negative test patch defines a schema that is enough 22:27:42 mtreisnish: Thanks: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/nova-api-validation-fw,n,z 22:27:43 So if we could agree on a sufficient schema that would be great 22:27:55 I'll promise to look at it in depth once we get the queue back in shape 22:28:09 And new apis should spec them, and have a way to fetch 22:28:17 dkranz: yeah I need to take another look at things too 22:28:40 ken1ohmichi: yeah you're right that is a lot of patches in flight 22:28:47 I'd like to get all the feedback and then we can submit another patch 22:29:05 dkranz: well I'd start with splitting the files out 22:29:16 So we will split the files and remove the JSON in the names 22:29:17 it's probably pretty close to ready then 22:29:32 we can fix things and iterate after we get an initial working version in 22:29:38 mtreinish: ok, great 22:29:59 ken1ohmichi: by the way, nice job on doing all the api validation bits in nova. I think that will be super helpful for the project. 22:30:23 dkranz: ok is there anything else on this? 22:30:23 I think that's all for that blueprint 22:30:29 sdague: Thanks. and thanks for reviewing:-) 22:30:34 ok then let's move on 22:30:42 #topic nova v3 xml removal 22:30:46 sdague: this is you again 22:31:04 oh right 22:31:12 so after polling lots of folks 22:31:17 a few email threads 22:31:38 it looks like we have concensus to pull xml out of nova v3 before icehouse 22:31:42 #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/025290.html 22:31:49 russellb just sent an email on it 22:32:07 so step one is to drop the tempest tests 22:32:16 https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/remove-v3-xml-api,n,z 22:32:25 so then we can pull out the nova pieces 22:32:32 huzzah. 22:32:40 Can I do it? 22:32:46 yes I'm very happy about this 22:32:50 :) 22:32:51 dkranz: I think sdague has dibs 22:32:53 sdague: I'm really glad that. 22:32:57 mtreinish: ok 22:33:06 yeh, I have dibs on the removes in nova. I'll start working on them next week 22:33:11 heh 22:33:17 sdague: I guess since you put them in ... :) 22:33:38 but if you could look over the tempest patches, a quick +2 / +A on those would be good 22:33:42 sdague: you shooting for top spot in "most lines removed" ? :) 22:33:52 it also saves us 5 - 10 minutes on the tempest-full runs 22:33:58 which would be helpful 22:33:58 russellb: he loves stat padding 22:34:02 heh :P 22:34:06 you should see his e-r patches 22:34:10 I just love killing xml 22:34:17 removing code is the best 22:34:21 agreed 22:34:44 I also think that this is going to simplify a lot of things long term 22:35:10 sdague: Is there a tc vote about this? 22:35:16 dkranz: it's coming 22:35:22 sdague: ok, good 22:35:28 I have a todo to work with markmc on language 22:35:32 tc may make a recommendation like ... "XML not required" 22:35:38 right, exactly 22:35:52 TC wouldn't be the one to make a call on the nova decision, for example 22:36:02 unless the TC wanted to put in some sort of cross project requirement for XML 22:36:03 russellb: Sure 22:36:04 and trove and ironic look like they will probably not do an XML api because of that 22:36:05 but that'd be bonkers 22:36:30 How about other project? They are volume, identity, network? 22:36:31 this is mostly the TC saying that we want to make sure all projects realize we only require a JSON api, not both 22:36:37 Based on their client, ceilo did xml but in a way that was auto-gen with json 22:36:38 +1 22:36:59 dkranz: it will be project decisions as what they do in the future 22:37:03 I think that is a viable approach 22:37:06 sdague: Sure 22:37:13 but this will give projects more flexibility to just not 22:37:29 ceilo did it with wsme, which has it's own set of challenges 22:37:36 they are free to continue down that path 22:37:43 sdague: yes,just pointing out that ceilo is supporting xml but without all the extra work, or without most of it 22:37:56 still have the verification and doc work ... 22:38:00 dkranz: sure, they have a very small api 22:38:06 and, like russellb said 22:38:09 russellb: Yes, and xml bigots will see it is not good xml 22:38:16 heh 22:38:32 dkranz: right, crappy xml isn't what anyone wants, not even xml people :) 22:38:46 anyway, moving on, I think 22:38:51 ok sure 22:38:56 * russellb ready for the flames 22:39:03 #topic Neutron testing 22:39:24 mlavalle: you have an update on api testing listed on the agenda 22:39:44 Most of the items in the api testing analysis have been claimed by test developers 22:40:06 at this point in time we have a community of 8 to 10 debs writing api tests 22:40:23 I train some of them last week in Montreal 22:40:50 mlavalle: hi . we are working on some est development . will send you for review 22:40:57 so I think we are in very good shape in this effort 22:41:12 mlavalle: ok are we still blocked by neutron gate issues on this? 22:41:26 as far as I know, yes 22:42:00 mlavalle: ok, I'll have to circle back with salv-orlando on helping him with that 22:42:03 over the next few weeks I will focus on reviewing the code that this groups submits 22:42:14 mlavalle: great 22:42:32 thanks again for training at that table in montreal, very helpful 22:42:33 once I deem it good enough, I will need help from core reviewers to help us merge the code 22:42:56 mlavalle: sure, but we'll need to get the neutron gate working again before we start approving things 22:43:39 onc the gate starts working again i'll ping you guys directly to get the patch sets merged 22:43:58 mlavalle: ok, are you guys using the same bp to track this 22:44:02 so there is one branch in gerrit? 22:44:16 no, we are using an entherpad 22:44:29 but I can create a blueprint if that helps 22:44:48 mlavalle: I just meant for submission if there is a branch view to see all the proposed api tests at once that helps 22:44:59 I guess it doesn't have to be a bp to do that 22:45:06 ok cool 22:45:22 mlavalle: ok is there anything else on this topic? 22:45:25 that's all I have 22:45:31 ok then let's move on 22:45:37 #topic Bugs 22:45:52 ok does anyone have any critical bugs to bring up 22:45:54 I have completely not looked at the bug tracker in 2 weeks 22:45:57 or bugs that need attention 22:46:07 sdague: I was looking a bit last night 22:46:12 and picked off a couple of simple ones 22:46:19 cool 22:46:31 I will spend some time on this tomorrow 22:46:49 I've been swamped with other things this week so far 22:47:12 ok I guess there's nothing to discuss on this one. Everyone's been kinda of busy and hasn't had much time to look at bugs 22:47:26 let's move to the next topic then 22:47:37 #topic Critical Reviews 22:47:51 does anyone have any reviews that they think need some attention? 22:48:10 to appease sdague preferrably that fix gate bugs :) 22:48:34 well, I'll point at the xml v3 ones 22:48:42 as it will shorten tempest runs, which will help 22:48:48 sdague: yeah I'll take a look after the meeting 22:48:49 mtreinish: There is my patch to turn on dumping of all errors into the console log 22:48:54 https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/remove-v3-xml-api,n,z 22:49:07 https://review.openstack.org/#/c/66190/2 22:49:07 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/remove-v3-xml-api,n,z 22:49:19 dkranz: can we actually put that into a separate file? 22:49:22 sdague: I will review them today. 22:49:26 #link https://review.openstack.org/#/c/66190/2 22:49:27 https://review.openstack.org/#/c/64447/ 22:49:35 in reading the neutron jobs they kind of overwhelm things 22:49:51 sdague: That is why I didn't do it in the first place 22:49:59 dkranz: oh... also, the error enforcing is actually off again 22:50:10 sdague: How come? 22:50:13 because we had a few more creep in, so we turned it off on sat 22:50:15 mtreinish: I'd like to get some attention on this patch https://review.openstack.org/#/c/66384/ and more generally know what's happening with the config-verification bp 22:50:17 #link https://review.openstack.org/#/c/64447/ 22:50:19 it was causing a bunch of resets 22:50:38 I have a change up to add at least some of those to the whitelist 22:50:40 rahmu: oh yeah that's a +A 22:50:46 there are some heat and conductor issues that cropped up 22:51:03 I figured lets get the gate back to something sane, then we turn it back on 22:51:07 Can't we just fix the issues? 22:51:08 rahmu: as for the bp I've got the rework patches up for review 22:51:19 https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/config-verification,n,z 22:51:20 I mean the new ones? 22:51:23 dkranz: with a 55 hr gate queue... not easily 22:51:27 mtreinish: thanks. Reviewing now. 22:51:32 sdague: Sigh 22:51:47 rahmu: you can base the swift ones on top of that after your patch merges 22:51:57 and the conductor one wasn't obvious, I hit up dansmith before he went on vacation 22:51:59 mtreinish: yes, that's my intention. 22:52:05 he looked for an hour, and couldn't figure it out 22:52:18 has to do with workers and how they talk to rpc 22:52:49 sdague: ok 22:53:32 ok are there any other reviews that people would like to bring up? 22:53:59 ok then lets move on to the open discussion 22:54:04 #topic Open Discussion 22:54:13 I just had one question 22:54:18 in the last ~6min does anyone have any other topics to bring up? 22:54:20 dkranz: ok 22:54:27 Has any one thought about making a pip package for tempest? 22:54:53 dkranz: I have thought about uploading to pip before 22:55:00 but we don't really do versioning in tempest 22:55:01 The assumptions about working directory with testr/discover are not friendly to that 22:55:23 I'm also not sure there is much benefit to doing it 22:55:41 mtreinish: ok, I'm not sure either 22:55:51 https://review.openstack.org/#/c/65805/1 Is it planned to restore the concurrency to 4 ? 22:55:57 dkranz: what do you mean about the working directory? 22:56:09 afazekas: not until we can convince ourselves that we can run under that load 22:56:22 mtreinish: pip install would put the code in the system dir somewhere 22:56:26 russellb and I have been adding various stats collection to devstack to figure that out 22:56:34 mtreinish: And then you can't discovre the tests 22:56:42 dkranz: related: some parts of tempest could be reusable for other projects. I'm thinking mainly about the rest_client. Has anyone thought about separating it into a reusable component? 22:56:58 but there was a strong feeling that we were driving so much load on the nodes that we were getting new classes of fails, and part of the stability issues 22:57:03 rahmu: That is a different issue and that was discussed recently 22:57:14 rahmu: But we don't want to take that on now 22:57:17 dkranz: any log on that discussion? 22:57:18 dkranz: oh, yeah you could solve that pretty easily actually. 22:57:28 for instance the tempest nodes would be at a load average of 8+ for 10s of minutes 22:57:44 mtreinish: How? I am not sure of those mechanics 22:58:07 rahmu: somewhere in the irc/qa logs I think 22:58:27 I just want testr to actually tell me what's wrong with a python file when it blows up, instead of doing super cryptic header dump 22:58:41 dkranz: well tempest will get installed into a tempest namespace so you can just make a tempest command that would import test_discover package 22:58:44 and run testr 22:58:44 dkranz: okay thanks. Will try to look it up. 22:59:05 mtreinish: ok, thanks. I;ll think about that. Not sure it's worth it either 22:59:43 ok so we're out of time 22:59:47 let's end for today 22:59:48 bye all 22:59:50 sdague: looks like with 2 thread the jobs takes longer 22:59:50 #endmeeting