17:02:48 #startmeeting qa 17:02:49 Meeting started Thu Sep 12 17:02:48 2013 UTC and is due to finish in 60 minutes. The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:02:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:02:52 The meeting name has been set to 'qa' 17:03:00 ok, who's around for the QA meeting? 17:03:02 Hi folks :) 17:03:02 hi 17:03:07 hi 17:03:16 hi 17:03:18 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting - agenda 17:03:30 hi 17:03:30 ok, first topic 17:03:50 #topic Welcome mkoderer and giulivo to tempest-core 17:03:52 :) 17:03:58 :) 17:03:59 welcome 17:04:02 afternoon 17:04:14 thanks for all the great work guys, very happy to have you part of tempest-core 17:04:20 thank you very much - I am very happy 17:04:27 hi 17:04:28 welcome mkoderer: 17:04:41 yes, great work guys 17:04:46 indeed 17:04:47 giulivo did such a great job, I am happy for him too 17:05:25 you guys should have all the permissions you need now, if you have any questions on things feel free to just put it out on #openstack-qa 17:05:44 I already have my first +A 17:05:45 ok, next topic 17:05:48 mkoderer: nice :) 17:05:57 have = gave 17:06:04 #topic Summit planning (sdague) 17:06:20 so mostly I just wanted to set up some time frames here 17:06:49 while the summit website is open for submissions, I don't intend to really have us look at them or start organizing until Oct 17:07:04 so probably the first week of Oct we'll put summit content on the agenda 17:07:15 sdague: makes sense 17:07:37 between now and then if interesting ideas come up that you think we should consider, please feel free to file it. But we'll hold until Oct to really make decisions there 17:07:37 sdague: just curious how many slots do we have this time? 17:07:41 we have 13 17:07:53 they are spread out through the week in 3 blocks 17:08:10 in a room where Infra will be, so we won't conflict this time 17:08:17 sdague: :) 17:08:43 ttx created a more broken up schedule, which I think is really good 17:09:00 sdague: Is there a link to it somewhere? 17:09:33 let me ping him to see if he has a public version of it, I just have a google spreadsheet link which I'm not sure he intended to be public 17:09:51 I'll also see him next week, I'll ask him there 17:10:06 ok, last administrative topic before content 17:10:11 #topic Leader for next week's meeting - sdague on a plane back from LinuxCon during this time 17:10:20 * jaypipes sad he won't be at summit ;( 17:10:23 :( 17:10:24 can I get a volunteer to run the meeting next week 17:10:26 no winky.. 17:10:28 jaypipes: bummer 17:10:32 yeah... 17:10:35 budget poop. 17:10:35 sdague: I can do it again 17:10:51 mtreinish: +1 17:10:53 jaypipes: that sucks, I'm not sure I can go either 17:10:54 #info mtreinish to run openstack-qa meeting next week 17:11:06 ok, thanks mtreinish 17:11:11 * jaypipes will be there in spirit :) 17:11:13 :) 17:11:46 it actually brings up the question of an intersession qa meetup, which we should maybe consider later. 17:12:03 ++ 17:12:14 but maybe we bring that up after we get through the rest of the agenda 17:12:22 yes, sorry for tangent 17:12:27 no worries :) 17:12:32 #topic Blueprints (sdague) 17:12:49 ok, I'm still catching up from being out, are there updates to blueprints people would like to report on? 17:13:08 I'll try to get things cleaned up from LinuxCon next week from the show floor as well 17:13:19 sdague: The blueprint about client library stability has basically been implemented by mordred. 17:13:31 sdague: Not sure if it has merged yet. 17:13:37 dkranz: cool, is it landed, or are there reviews that need eyes? 17:14:05 sdague: I think it is waiting on some other devstack change. I can dig up the patch link after the meeting. 17:14:16 dkranz: ok, that would be great to make happen before Havana releases, so if we know of reviews that need eyes on that, please let me know 17:14:18 great 17:14:21 sdague: But it was reviewed favorably 17:14:27 any other blueprints? 17:14:28 sdague: Will do 17:15:02 I will have to say that everyone owes mtreinish a beer over getting parallel tempest into the gate 17:15:10 that was a huge win last week 17:15:26 I don't think we'd have survived feature freeze day without it 17:15:31 sdague: Yes, very much so. 17:15:53 sdague: It is interesting that even that slight stress has uncovered a lot of things. 17:15:56 It's pretty awesome that tempest tests now take only slightly longer than nova unit tests :) 17:16:06 +1 beer to mtreinish :) 17:16:06 ++mtreinish 17:16:17 thanks everyone, I'll always accept free beers :) 17:16:24 :) 17:16:26 \quit 17:16:27 :) 17:16:30 yep, the test env has definitely started to act more like real systems, which is very good. shook out a bunch of new races. 17:16:48 ok, anything else on blueprints? 17:17:05 mlavalle: you around? 17:17:12 sdague: yep 17:17:24 #topic neutron testing status (mlavalle) 17:17:40 ok, you have the floor, where do we stand on neutron? 17:17:51 sdague: so, I've been working over the past few days in adding network isolation to enable parallel runs 17:17:55 https://review.openstack.org/#/c/45578/ 17:18:24 great, lets get eyes on that review 17:18:45 mtreinish: what do you think? are we good to test it? 17:18:55 mtreinish, especially as you did the last refactor there, I'd like your opinion on it 17:19:14 mlavalle, sdague: ok I'll take a look at the latest revision 17:19:18 great, thanks 17:19:27 testing it is fun because to enable it to actually run requires a devstack change 17:19:33 which can't happen with this patch :) 17:19:42 mtreinish: ok, what's the devstack change we need? 17:19:50 and is that proposed anywhere? 17:19:56 sdague: tenant isolation is disabled if neutron is enabled 17:20:06 sdague: not yet I was going to put it together after the meeting 17:20:12 mtreinish: ok, cool 17:20:26 mtreinish: we can add second change just for testing based on this which reads the config in a different way :) 17:20:30 maybe we need another non-voting job on neutron that sets that 17:21:05 afazekas: yeah that would work too (we could just hard code it to true) 17:21:10 much like we did with testr initially 17:21:36 any volunteers to make the new devstack gate job for that? 17:21:57 sdague: I'm not sure that's really needed (unless you're talking about a neutron parallel run) 17:22:03 sdague: The preferred way (by infra) to do this now is by using the experimental queue in zuul. 17:22:22 mtreinish: that's what I mean, neutron parallel job 17:22:49 sdague: Instead of creating non-voting jobs 17:22:50 dkranz: ok, I haven't looked at the experimental queue 17:22:54 dkranz: ok 17:23:01 sdague: That's what I did for heat. 17:23:07 I'll have to catch up on that bit, sounds cool 17:23:15 ok, anything else on neutron testing? 17:23:26 sdague: once we get that running, I'll get back to https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full. We've made good progress fixing issues, so this weekend i'll send an email to the ML 17:23:35 what are still our blocking issues on full gate 17:23:41 ok, just what I was about to ask :) 17:23:46 great 17:23:51 sdague: Some one is working to make all the tests on stable/grizzly work. 17:23:56 with an assessment of how close we are to be done 17:24:08 sdague: Do we want to try to turn on the full gate there once that is done? 17:24:08 #info mlavalle to send status update on full gate for neutron 17:24:29 that's all I have 17:24:30 dkranz: honestly, stable/grizzly is the past, I'm not interested in fixing it until master is fixed 17:24:30 sdague: I mean all neutron (quantum) tests 17:24:42 sdague: He is fixing both 17:24:44 ok 17:24:57 The question is whether to turn on the full gate in grizzly once he is done. 17:25:18 if it works, I think that's fine 17:25:38 sdague: OK, I'll tell him we would like to do that and to keep us posted on progress. 17:25:47 sounds good 17:26:03 ok, next topic 17:26:08 #topic whitebox test removal (sdague) 17:26:27 sdague: Wasn't it done already :) 17:26:34 sdague: https://review.openstack.org/#/c/46116/ 17:26:37 so I think we came to a general concensus on irc the other day to pull whitebox and add nova unit tests 17:26:54 dkranz: well that's only 1/2 of it :) 17:27:16 sdague: Less than half, work-wise. 17:27:20 because we still need a volunteer to actually do the unit test enhancements on nova so that we cover it 17:27:22 there is pending keystone whitebox test change on gerrit as well 17:27:52 sdague, i can volunteer to take a look on what is covered in nova. 17:27:59 great 17:28:23 #info adalbas to port whitebox tests into nova unit tests 17:28:31 Normally I would say wait to remove whitebox but it is breaking the nightly build. 17:28:51 afazekas: so I think we saw with all the whitebox tests that what they were testing was doable in the unit tests by the projects 17:29:04 dkranz: yeah that's why I removed it, because it doesn't do anything right now except generate errors 17:29:07 and because the db schemas changed so fast, tempest was really not the best place for them 17:29:19 enhance it closer to the projects 17:29:41 sdague: I think we all agreed on that. 17:29:44 yep 17:30:08 ok, so my suggestion is -2 any new whitebox patches that come in if you see them, on that kind of direction 17:30:21 unless there are objections 17:30:32 sdague: Should send to the ml as well. 17:30:39 yes, I'll send to ML 17:31:05 #action sdague to send ML email about change in approach for whitebox testing 17:31:37 ok, next topic 17:31:51 mkollaro: you around/ 17:32:06 sdague: She is sick today I just found out. 17:32:09 ok 17:32:23 so let's skip that and just do reviews, then open discussion 17:32:30 #topic Critical Reviews (sdague) 17:32:41 ok, critical outstanding reviews that we need eyes on? 17:32:49 links appreciated 17:32:58 sdague: This is the client lib stability one https://review.openstack.org/#/c/41931/ 17:33:17 great, up in a tab for me, I'll check it later 17:33:22 other ones? 17:33:27 this one from afazekas https://review.openstack.org/#/c/35165/ 17:33:54 it might fix a lot of rechecks related to the parallel runs 17:34:09 ok, great 17:34:36 afazekas: did we land the client changes to support that already? (I remember reviewing it, but I don't remember if it merged) 17:34:58 I'll review that later today as well 17:35:05 and other eyes would be good 17:35:06 sdague: the xml version got an approval from you 17:35:12 afazekas: great 17:35:34 afazekas: yeh, I just couldn't remember if I was the first or second +2 :) 17:35:42 :) 17:36:03 i think it got merged 17:36:06 other reviews people think we need extra eyes on? 17:36:46 sdague: I'm here 17:37:00 oh, hey, we can do your topic now 17:37:13 #topic OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro) 17:37:13 sure 17:37:18 mkollaro: you have the floor 17:38:11 we want to share the test plans for new features with upstream, I created a repo for them https://github.com/mkollaro/openstack-testplans 17:38:59 I was wondering what it would take to get it put under some openstack managed group, so it wouldn't be just mine project and people from the community could contribute to it 17:39:08 *my 17:39:35 an example of a test plan is here https://github.com/mkollaro/openstack-testplans/blob/master/glance/multilocation.rst 17:40:02 they would be linked in the blueprints 17:40:05 sdague: This was an attempt to address the issue of not enough info in blueprints 17:40:26 so is there a reason not to use the wiki for these? 17:41:04 I guess that's my only question, the plusses / minusses of doing this in git vs. the openstack wiki 17:41:38 that would be another option...but AFAIK devels don't like to use the wikis too much and I'd like them to work with us around this more 17:42:10 handling test plans as code is better IMHO, since it pretty much _is_ code 17:42:12 I also understand that while the setup is manual, is the intent to turn the actual plan into something like a scenario test (post setup) 17:42:15 can we add in whiteborad of blueprint so that all in one place 17:42:43 blueprints don't have formatting and nobody is using them 17:43:09 it seems very hard to get devels to use the whiteboard in blueprints 17:43:43 I hoped that the combo of git+rst text files would make contributions more likely 17:43:59 is there a way to links such a test cases in a blueprint? 17:44:13 mkollaro: I guess this raises the idea about whether if we enhanced some of the markup in the scenario tests if these could be just scenario tests in tempest, plus a preamble that explains the setup 17:44:55 mkollaro, is the point of this being tracking the existing cases or both the existing and wanted cases? 17:44:57 mkoderer: AFAIK there is no special field for it, but you can paste the link on the whiteboard 17:44:59 because our community is very much about automation, so I feel like just documents for test execution doesn't deliver on the idea of repeatability, though I definitely get the need to have more details 17:45:35 mkollaro: ok sure a special field would be nice :) 17:46:06 sdague:+1 17:46:55 Newbie here. There may be easy to make wiki test cases executable but it would need to be a project 17:47:04 sdague: I'm not sure if they should be used only for scenario tests and I'm primarily intending them as a means of communication, not documentation 17:47:29 giulivo: not the existing ones, there are almost 1k test cases, that would be impossible 17:47:54 mkollaro: the problem is test plans that aren't actually test code aren't things we can gate on 17:47:56 how is this: blueprints map ot plans, a blueprint also has a link to descriptive wiki, but cases are tracked in the blueprint todoitems? 17:48:08 I think what Sean is asking is why not just provide code that works, rather than pseudo-code instructions? 17:48:20 dkranz: right exactly 17:48:25 sdague: yes, but I'm afraid there is no way to do that :( 17:48:41 I could imagine that document could actually be a scenario test, with extensive markup in it 17:48:52 mkollaro: why not? 17:49:33 before we get to executable code, we need to figure out a test plan...I spent hours of talking to the devel before I wrote that one I linked 17:49:56 after we got code, the test plan can be pretty much removed as far as I care 17:50:01 Sdague: yes. That is what cucumber test tool does 17:50:40 mkollaro: so if it's transient like that, then I think etherpad or wiki is better than git 17:50:55 because then we can mark it as done once it's landed in code 17:51:10 that's why I wanted to use the todoitems 17:51:27 the problem is, nobody really writes down some expected behavior of a feature or even some basic description...this could help with that 17:51:52 Mkollaro:why not submit plan as big to be executed and reviewed? 17:51:53 rockyg: right, but we already have a test framework, so while I like cucumber in theory, I think in practice it's not the right direction for us right now 17:52:18 Sorry. Bug. 17:52:20 mkollaro: so I think I understand the problem, but I'm not sure another git repo is going to solve it. 17:52:37 sdague: wikis wouldn't be so easily parsable, there is this idea of creating some replacement of our (redhat) current test plan system or upload it there with some script 17:53:41 mkollaro: ok, how would you expect this would become part of OpenStack validation eventually? 17:53:42 this could be easily doable with the rst format, if we ever want that 17:53:56 sdague: what do you mean by validation? 17:54:16 the gate, or other tools we make available to people to verify openstack is working 17:54:41 sdague: I never meant this as a validation thing, more like a communication channel 17:55:29 sdague: I think what we want here is a way to extract information from a developer and "add it to the blueprint" in some other way than literally doing that. 17:56:02 ok, so I feel like in OpenStack our culture is that we communicate with code, which is why we don't just have HACKING.rst, but actually executable hacking checks 17:56:25 I think this is probably a good design summit topic 17:56:36 sdague: yeah, but we need to figure out first how to write the test, typically together with the devel 17:56:39 sdague: Yes, but if I talk to a dev and take notes, code does not come out right away. ANd some one else might right the actual code. 17:56:53 write 17:57:34 We are not going to solve this in our remaining two minutes 17:57:40 agreed 17:58:04 ok, so should we take it to the mailing list? or do we want to let this simmer until summit? 17:58:17 but in case I decided to use wikis, where could I put the test plan? 17:58:40 mkollaro: we'd make a namespace in the OpenStack wiki for them. 17:59:09 sdague: Can any one do that? 17:59:12 so in our last 2 minutes, mostly I want to figure out where we want to continue the conversation 17:59:17 dkranz: yep 17:59:25 I don't really see why a wiki would be an improvement over git+rst, but if you really prefer that, sure 18:00:16 mkollaro / dkranz: want to take this to the ML? 18:00:28 How about proposal on ml to flesh out topic for summit? 18:00:48 sdague: OK, we'll think about it. 18:01:19 cool, sounds good. whatever forum you want to, I think there are interesting ideas here, just need to explore it more 18:01:26 ok, with that we're over, so I'll end up things 18:01:30 #endingmeeting 18:01:35 #endmeeting