22:02:35 <mtreinish> #startmeeting QA
22:02:36 <openstack> Meeting started Thu Dec 12 22:02:35 2013 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:40 <openstack> The meeting name has been set to 'qa'
22:02:59 <SergeyLukjanov> o/
22:02:59 <mtreinish> hi who's here today at our new time?
22:03:04 <dkranz> o/
22:03:05 <maurosr> \o
22:03:11 <masayukig> hi
22:03:11 <mlavalle> I am
22:03:14 <giulivo> hi
22:03:22 <rahmu> o/ first time
22:03:29 <cyeoh> hi!
22:03:38 <giulivo> cyeoh !! :)
22:03:40 <cyeoh> first time for me too :-)
22:03:41 <sdague> oh right -5
22:03:48 <mtreinish> today's agenda: https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
22:03:52 <sdague> I really thought this was another hour :)
22:03:53 <cyeoh> guitarzan: :-)
22:03:59 <mtreinish> sdague: yeah I was thrown too I assumed the same thing :)(
22:04:08 <cyeoh> its kind of nice having it straight after the Nova meeting
22:04:22 <sdague> yeh
22:04:27 <mtreinish> let's get started then
22:04:32 <mtreinish> #topic Blueprints
22:04:49 <sdague> so giulivo has done an awesome job on clean up here
22:04:57 <mtreinish> giulivo: since you're taking the charge of blueprint clean up is there anything to report?
22:05:14 <giulivo> sure so firstly, please not I sent you an email with comments about a few bp
22:05:34 <giulivo> *note
22:05:46 <giulivo> unless you disagree those will be closed/approved accordingly to the comments in the email
22:05:56 <giulivo> if you disagree, please reply
22:05:57 <sdague> giulivo: sounds great
22:06:16 <sdague> giulivo: there are also a ton of "new" blueprints at the bottom
22:06:30 <sdague> you going to purge those as well?
22:06:31 <mlavalle> giulivo: is there a deadline to reply to the email?
22:06:47 <giulivo> well this was a topic I'd wanted to discuss
22:06:55 <sdague> I think anything in new & unknown state should just be marked invalid
22:07:14 <giulivo> some are basically asking for more tests to cover one or another functionality, what shall we do with these? ideas?
22:07:38 <giulivo> I'd close these too
22:07:49 <mtreinish> giulivo: so basically how do we track new tests?
22:07:57 <mtreinish> or new test development
22:08:04 <sdague> giulivo: so if people are actually targeting them to milestones, I think we can leave them
22:08:05 <dkranz> or new anything
22:08:06 <giulivo> some instead are asking for new tests to cover components we don't have the services for and I'd actually approve these
22:08:34 <giulivo> mtreinish, indeed that was my point of concern, I think the easiest way for that is just a wiki page, maybe organized by component
22:08:57 <mtreinish> cyeoh: well you said a google doc spreadsheet works well for this right?
22:09:01 <giulivo> I can't think of anything else, but I'm obviously open to input
22:09:08 <cyeoh> yea a spreadsheet works really well for api tests
22:09:29 <sdague> I guess I'd take a step back
22:09:37 <cyeoh> if we really want to we can have a blueprint point to the appropriate spreadsheet. But a bp is in general a really cumbersome way of tracking progress for that sort of thing
22:09:43 <giulivo> but wouldn't that force people to have a google account? what would be the benefits compared to the wiki?
22:09:47 <sdague> basically: what is a blueprint useful for
22:10:07 <dkranz> sdague: Avoiding duplication
22:10:11 <sdague> it's useful if a person is committing to deliver a thing by a milestone so we can ensure it gets review eyeballs
22:10:33 <sdague> dkranz: sure, but that again needs both an owner and a milestone
22:10:44 <dkranz> sdague: Yes.
22:10:48 <sdague> and regularly updated status on it
22:11:01 <dkranz> If we simply insisted that new blueprints had an owner and milestone it would to a long way
22:11:09 <dkranz> go
22:11:39 <sdague> and the reason we go to the effort of that is to ensure that we prioritize reviews for it
22:12:20 <sdague> and if you have a blueprint, I also expect you to come to this meeting, or provide regular updates
22:12:23 <sdague> in some other way
22:12:53 <sdague> so I think doing the giant purge that giulivo is doing is great
22:12:56 <giulivo> sdague, I totally agree with that, but I still wouldn't apply this to "we need tests for cinder backup" type of blueprints
22:13:05 <sdague> giulivo: agreed
22:13:25 <sdague> especially as there are already at least 3 bugs I duped together on that today :)
22:14:08 <sdague> ok, giulivo I think you can probably triage out 75% of the blueprints with the feedback you gathered so far right?
22:14:16 <giulivo> yes
22:14:17 <sdague> then maybe next week we cycle again on what's left
22:14:30 <sdague> to handle any sticky issues around them
22:14:45 <giulivo> but one thing, probably not on topic, remains open and that is how we track the actual tests people want to add
22:14:55 <cyeoh> maybe we should also write it down in a wiki what sort of things we want blueprints for, and what we don't (and how people should handle it instead)?
22:14:58 <sdague> giulivo: can we circle to that at the end?
22:15:10 <giulivo> sdague, indeed not on topic, agree on that
22:15:16 <sdague> cyeoh: so, I'm actually becoming more a fan that we should do that in the tempest docs tree
22:15:16 <mtreinish> sdague: it is the last topic on the agenda :)
22:15:21 <sdague> mtreinish: great :)
22:15:28 <sdague> ok
22:15:33 <cyeoh> sdague: yep, that'd be fine with me
22:15:39 <sdague> #action giulivo to do the great blueprint purge
22:15:52 <sdague> and there was much rejoicing!
22:16:20 <mtreinish> ok is there anything else that needs to be discussed about blueprints?
22:16:29 <mtreinish> otherwise let's move on to the next topic
22:16:37 <giulivo> mlavalle, one thing
22:16:55 <giulivo> I noticed a blueprint where we suggest to have tests for different network topologies in neutron
22:16:57 <mlavalle> giulivo: listening...
22:17:03 <giulivo> I wonder if that is at all feasible with our infra?
22:17:31 <giulivo> https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api
22:17:39 <mlavalle> I think it is…. the question is to prioritize that development with thereat of things I am doing for Neutron
22:17:40 <sdague> giulivo: I think we should gather more details on that, it's really more of an openstack-ci item
22:18:13 <mlavalle> we can talk after the meeting
22:18:17 <mtreinish> mlavalle: yeah I think right now there are other priorities for neutron testing
22:18:26 <giulivo> ok for me
22:18:29 <mtreinish> which is a good segway into the next topic
22:18:38 <mtreinish> #topic Neutron testing
22:18:48 <mtreinish> mlavalle: you're up
22:19:05 <mlavalle> so in the api testing fron I didi 3 things this week
22:19:39 <mlavalle> number 1 I created a wiki page with a How To for API tests development for Neutron https://wiki.openstack.org/wiki/Neutron/TempestAPITests
22:20:15 <mlavalle> number 2 I sent a message to the ML recruting developers and pointing to the wiki page
22:20:53 <mlavalle> number 3 I kept adding to the gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron
22:21:19 <mlavalle> at this point I have completed L2, L3, extensions management and provider networks
22:21:43 <mlavalle> I will keep going through the API spec and hope to be finished by next week
22:21:59 <mlavalle> at that point I will just start developing tests
22:22:09 <mlavalle> from the list
22:22:38 <mtreinish> mlavalle: ok as we were talking about a min. ago it might be useful to put that list into a spreadsheet somewhere
22:22:54 <mtreinish> that seems to work really well for splitting up api tests
22:23:07 <mlavalle> sure…… is that a Google spreadsheet?
22:23:13 <mtreinish> especially because it's getting kind of length
22:23:52 <anteaya> let's also acknowledge that EmilienM's patch merged and neutron grenade tests now run, though they don't yet test anything
22:23:55 <mtreinish> mlavalle: sure I guess, I don't think we have openstack infra cloud spredsheet
22:24:04 <cyeoh> mlavalle: yea we've used a google spreadsheet in the past, but anything really like that would be fine
22:24:23 <cyeoh> this is an example: https://docs.google.com/spreadsheet/ccc?key=0AmYuZ6T4IJETdEVNTWlYVUVOWURmOERSZ0VGc1BBQWc#gid=0
22:24:31 <mlavalle> cool…. I'll move it to a spreasheet
22:24:39 <cyeoh> it allows for really fine grained self allocation of work
22:25:24 <mlavalle> any other questions / observations on this regard?
22:25:44 <sdague> anteaya: yep, thankful for EmilienM's work there
22:26:05 <sdague> what about the SSH bug? Anyone have any news on that
22:26:16 <anteaya> we need some devs coming forward to write some of those tests on the list
22:26:24 <anteaya> so far, no volunteers
22:26:29 <anteaya> so if any happen by
22:26:40 <anteaya> point them to -neutron and to mlavalle and myself
22:26:48 <anteaya> they don't need to commit much
22:26:53 <anteaya> any help appreciated
22:27:02 <anteaya> salv-orlando just came online
22:27:07 <anteaya> and has 2 hours to dig into it
22:27:24 <anteaya> I will give him his time and get a report from him before he goes offline
22:27:36 <anteaya> hopefully to hand the baton to someone else
22:27:59 <anteaya> my problem is that there seem to be many error messages captured by that logstash fingerprint
22:28:09 <anteaya> so in my work I am having a hard time
22:28:16 <anteaya> but my logstash foo is weak
22:28:24 <anteaya> that is all I have on the ssh bug
22:28:42 <sdague> anteaya: thanks
22:28:58 <mtreinish> ok is there anything else to discuss on the neutron testing front?
22:29:45 <mtreinish> ok then let's move on
22:29:49 <mtreinish> #topic Bug status
22:30:01 <mtreinish> so adalbas said he couldn't make it today
22:30:25 <sdague> so, we started the day at 276 ?
22:30:30 <mtreinish> but he wanted to thank everyone who contributed to the bug triage day today (or yesterday for some people :)
22:30:50 <mtreinish> sdague: he put some notes up here: https://etherpad.openstack.org/p/tempest-bug-triage
22:30:53 <mtreinish> #link https://etherpad.openstack.org/p/tempest-bug-triage
22:31:09 <sdague> http://status.openstack.org/bugday/tempest.html - cron was broken earlier in the day, so it doesn't have full progress
22:31:20 <sdague> but I just got us to 97
22:31:23 <sdague> last check
22:31:30 * sdague still triaging
22:31:35 <mtreinish> wow that's a big drop
22:31:38 <sdague> so thanks much to everyone
22:31:57 <sdague> huge amount of work there, and it helps in getting patterns out of it
22:32:12 <sdague> like the fact that there are a ton of nova state transition bugs that get filed in piecemeal
22:32:44 <sdague> which makes me think we actually need a new tempest test(s) that just do large_ops style run the state engine and try to break nova
22:33:11 <mtreinish> sdague: something like the stress tests with a fake virt driver?
22:33:20 <mtreinish> sdague: we were talking about something like that in HK
22:33:56 <sdague> mtreinish: yeh, that would probably be a starting point
22:34:12 <sdague> honestly I don't know if these are nova-compute bugs, or virt layer bugs with libvirt
22:34:20 <giulivo> I would actually +1 the idea of using fake drivers for the api/gate tests and keep the real drivers for the periodic jobs
22:34:34 <sdague> giulivo: I think we need real drivers in the gate
22:34:38 <mtreinish> yeah if they're libvirt bugs we won't catch them with the fake driver
22:34:48 <giulivo> but at gate we're not testing libvirt
22:34:51 <giulivo> nor lvm
22:34:55 <dkranz> And we need to ssh
22:35:07 <sdague> giulivo: ?? we are testing those things today
22:35:21 <dkranz> We need the gate jobs to be "real"
22:35:21 <sdague> dkranz: so this class of bugs doesn't need ssh
22:35:32 <sdague> right, agreed, I think that's a distraction
22:35:38 <sdague> I'm not trying to remove anything
22:35:39 <dkranz> sdague: I was referring to the comment about gate jobs
22:35:43 <sdague> yep
22:36:10 <mtreinish> sdague: well I can push out a new jjb job for running stress with a fake virt driver for like 20min
22:36:14 <giulivo> heh okay I see I'm a minority here, will try to bring it up again differently not during the meeting :)
22:36:15 <sdague> staying on topic-ish, I think nova state bugs are huge class of issues today, and we should try to figure out how to make them more frequent
22:36:16 <mtreinish> that should be straightforward
22:36:32 <mtreinish> we can do it nonvoting to see what it turns up
22:36:38 <sdague> mtreinish: actually, I think we need to think through this further
22:36:48 <ken1ohmichi_> one question, can we get libvirt log on the gate?
22:36:50 <giulivo> but the thing is I don't think at gating we should actually be testing if libvirt behaves correctly, but if the api behaves correctly
22:36:58 <sdague> because I actually expect this might be very surgical
22:37:12 <sdague> ken1ohmichi_: you know where it's logging to?
22:37:18 <giulivo> libvirt is a subset of the nova drivers and , maybe just because it is the most common , we pick it for the "real" periodic jobs
22:37:40 <sdague> giulivo: I disagree :)
22:37:40 <ken1ohmichi_> sdaue: log files under /var/log/libvirt/
22:37:44 <mtreinish> giulivo: how about we save that discussion for after the meeting
22:37:54 <sdague> ken1ohmichi_: we could definitely add it
22:38:17 <ken1ohmichi_> sdague: thanks, will check the way.
22:38:28 <sdague> ken1ohmichi_: get with me in -qa after the meeting, and I'll give yuo the pointers as to where to do that
22:38:53 <ken1ohmichi_> sdague: great, will catch you:-)
22:38:58 <maurosr> one more thing related to the triage, I saw sdague and dkranz talking about it all day, so should we have guidelines to bug report? cause those tracebacks don't really help us, people are just trying to use bugs in tempest to rechecks
22:39:13 <sdague> maurosr: yes, definitely
22:39:26 <mtreinish> maurosr: yeah we need better triage guidelines
22:39:38 <mtreinish> I think that is something adalbas was planning to work on moving forward
22:39:51 <sdague> do we want to do that now? or discuss in a review? I was going to propose something into the docs tree about triage and good bugs
22:40:06 <dkranz> mtreinish: My patch to send non-whitelisted errors to the log on failure will help
22:40:17 <mtreinish> sdague: yeah I think doing it in a review would be fine
22:40:37 <dkranz> mtreinish: It will then be easy to get the error from the log and not just the backtrace, if there is one
22:40:37 <mtreinish> dkranz: the d-g one?
22:40:53 <dkranz> mtreinish: Yes, but I don't know how to write shell script so it is not working
22:41:05 <mtreinish> dkranz: ok I'll take a look after the meeting
22:41:05 <anteaya> will you post you "good bugs guidelines" to the ml? I need to read them
22:41:12 <anteaya> or a link to the ml
22:41:24 <dkranz> mtreinish: Help appreciated since I didn't understand Clark's comment
22:41:32 <sdague> dkranz: where's the review?
22:41:41 <clarkb> dkranz: I am happy to clarify :)
22:41:47 <mtreinish> sdague: https://review.openstack.org/#/c/61850/
22:41:56 <sdague> anteaya: I think we'll handle it in a review in tempest doc tree
22:42:03 <dkranz> clarkb: ok, I'll ping you later, thanks
22:42:06 <anteaya> sdague: I will look there
22:42:46 <mtreinish> We've still got 4 topics on the agenda, so is there anything else to discuss on bugs?
22:42:54 <dkranz> sdague: https://review.openstack.org/#/c/61850/1/devstack-vm-gate.sh,unified
22:43:21 <mtreinish> dkranz: a good segway :)
22:43:28 <mtreinish> #topic Critical Reviews
22:43:45 <mtreinish> so does anyone have any reviews that they would like to bring up
22:43:51 <mtreinish> that they think need attention
22:44:17 <dkranz> mtreinish: I proposed we give some priority to heat and ceilo reviews
22:44:24 <sdague> actually, the review queue is pretty short right now
22:44:27 <sdague> it's pretty nice
22:44:28 <mtreinish> dkranz: that's the next topic
22:44:34 <dkranz> mtreinish: ok :)
22:44:40 <mtreinish> I actually have 2
22:44:41 <rahmu> I have a question about this review: https://review.openstack.org/#/c/59759/
22:44:52 <mtreinish> #link https://review.openstack.org/#/c/60866/
22:44:58 <rahmu> which I guess could be added to the conf file cleanup bp
22:44:58 <mtreinish> and
22:45:01 <mtreinish> #link https://review.openstack.org/#/c/60578/
22:45:14 <sdague> rahmu: fire away
22:45:22 <rahmu> have we settled on a way to skip a test if a middleware (in the case of swift) is not installed?
22:45:32 <rahmu> there was some talks on the ml http://lists.openstack.org/pipermail/openstack-dev/2013-December/thread.html#21121
22:45:50 <rahmu> and sdague said that a decorator on setupclass would be okay
22:45:59 <mtreinish> rahmu: so I've been working on an approach with the extensions
22:46:14 <mtreinish> rahmu: actually using decorators for something that depends on a config variable isn't going to work
22:46:30 <mtreinish> you need to do it inside the function
22:46:59 <mtreinish> something I figured out a couple of days ago, I can give you hand with it after the meeting
22:47:03 <sdague> rahmu: so I'd say follow mtreinish's lead on how he's tackling the compute extensions
22:47:18 <rahmu> okay thanks. I'll ping you later mtreinish
22:47:40 <mtreinish> rahmu: ok cool
22:47:48 <sdague> mtreinish: +2 to both of your reviews
22:47:54 <mtreinish> sdague: sweet thanks
22:48:10 <sdague> https://review.openstack.org/#/c/61873/ - easy, and probably closes a bug :)
22:48:11 <mtreinish> ok, are there any other reviews anyone would like to bring up?
22:48:28 <mtreinish> #link https://review.openstack.org/#/c/61873/
22:49:03 <mtreinish> I'll +2 it, I just want to dig a little bit more in why it's there
22:49:08 <mtreinish> because it's a little strange looking
22:49:26 <giulivo> and I don't understand why it would be failing anyway
22:49:26 <mtreinish> ok, lets move on
22:49:28 <cyeoh> mtreinish: possibly someone was thinking of checking server status
22:49:51 <dkranz> mtreinish: I believe that line was recently added to *avoid* a race condition. I'll track it down.
22:50:02 <mtreinish> cyeoh: yeah, I had a couple of ideas
22:50:12 <mtreinish> #topic We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
22:50:15 <dkranz> cyeoh: That method now checks server status too I believe
22:50:23 <dkranz> cyeoh: As of a recent change
22:50:35 <dkranz> mtreinish: Any dissenters?
22:50:36 <cyeoh> dkranz, ah ok
22:50:36 <mtreinish> dkranz: this lengthy topic is yours...
22:50:46 <mtreinish> dkranz: no I'm fine
22:50:59 <mtreinish> my only concern is heat tests don't work and we have no way to verify them
22:51:14 <dkranz> mtreinish: Some of them do work.
22:51:23 <sdague> yeh, we're basically wedged on getting infrastructure up for the slow tests
22:51:23 <dkranz> mtreinish: I don't know what to do about the others
22:51:31 <sdague> I've been trying to review the other ones
22:51:38 <dkranz> sdague: right
22:51:51 <sdague> but I think that is fine as priorities go
22:52:05 <dkranz> sdague: At this point I think it would do more good than harm to review things even if they don't run. Just for this special case.
22:52:16 <sdague> dkranz: it would be good to get folks working on those into the -qa channel as well on a regular basis
22:52:25 <anteaya> is there a rep from heat and from ceilometer in this meeting?
22:52:26 <dkranz> Yes
22:52:29 <sdague> so we can give more specific feedback
22:52:33 <dkranz> stevebaker:  ^^^
22:52:51 <dkranz> I think we can move on
22:52:52 <sdague> stevebaker has been great
22:52:58 <sdague> but we need a ceilo person
22:53:08 <sdague> dkranz: can you take a todo to recruit one
22:53:09 <sdague> ?
22:53:20 <dkranz> sdague: Sure
22:53:43 <mtreinish> #action dkranz to recruit a ceilo core to be focal point on ceilo tempest tests
22:53:48 <mtreinish> ok let's move on
22:53:50 <sdague> thanks mtreinish
22:53:58 <mtreinish> #topic Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)
22:54:05 <mtreinish> dkranz: another lengthy topic
22:54:32 <anteaya> the topics passed pep8 apparently
22:54:52 <dkranz> mtreinish: This is really a question of whether we should all review everything scattershot or each take an area to hit first when reviewing
22:55:07 <dkranz> Doesn't mean we are limited to one area
22:55:22 <dkranz> But I found reviewing the nova v3 changes got easier after I had done a bunch
22:55:46 <sdague> so I was looking at some neutron tests recently
22:56:04 <sdague> and I asked in the review for a link to the API docs for that section
22:56:14 <sdague> which was provided in a comment by lifeless
22:56:19 <sdague> and it was incredibly useful
22:56:31 <cyeoh> well I'd agree there is definitely a big learning curve to be able to review changes for a new api/project
22:56:43 <mtreinish> sdague: so your saying you want links to api specs in commit messages for new tests now?
22:56:45 <sdague> so I kind of wonder if we should ask that of at least API tests
22:56:55 <sdague> mtreinish: or in a comment
22:57:07 <giulivo> sdague, I would put these links in the aforementioned wiki page were we track api tests
22:57:11 <cyeoh> sdague: I think that's a good idea, though admittedly for the v3 api we don't really have a spec document yet
22:57:14 <sdague> giulivo: or there
22:57:28 <sdague> cyeoh: yeh, so it's hard to validate API tests for an API with no docs :)
22:57:46 <sdague> so it seems like we've got a cart / horse problem there
22:57:55 <dkranz> sdague: I asked for that fot nova v3 and was pointed to a diff with v2 which was what was needed.
22:57:56 <anteaya> I think it is a reasonable request
22:58:00 <cyeoh> sdague: yea we only have v2/v3 diff document which is still buggy
22:58:13 <sdague> cyeoh: well as dkranz said, it was useful
22:58:47 <sdague> but, yeh, some easier way of pointing reviewers to the spec would just expedite reviews
22:58:53 <anteaya> exactly
22:59:04 <dkranz> I think the issues is that when we started, there were a few projects/apis and we all knew all of them
22:59:08 <cyeoh> agreed.
22:59:18 <dkranz> But I have not been able to keep up with all the new apis and projects
22:59:29 <dkranz> I was suggesting we divide and conquer for that
23:00:01 <giulivo> dkranz, maybe we could put a couple of nicks next to each component telling people to add those as reviewers *if* in need ?
23:00:05 <dkranz> But not in a rigid way. Let me think a little more and propose something
23:00:13 <mtreinish> dkranz: well, we're out of time
23:00:13 <dkranz> Not right now
23:00:15 <sdague> dkranz: cool
23:00:16 <giulivo> (I mean that in the tempest docs)
23:00:25 <sdague> yeh, I think we need to give up the room
23:00:29 <mtreinish> giulivo: sorry we couldn't get to your topic
23:00:34 <mtreinish> we'll save it for next week
23:00:37 <mtreinish> #endmeeting