17:00:24 <mtreinish> #startmeeting qa
17:00:25 <openstack> Meeting started Thu Dec 19 17:00:24 2013 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:28 <openstack> The meeting name has been set to 'qa'
17:00:32 <mtreinish> hi, who's here today?
17:00:37 <giulivo> me
17:00:47 <andreaf> hi
17:01:08 <mtreinish> today's agenda:
17:01:12 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:33 <krtaylor> o/
17:01:43 <mtreinish> ok well let's get started
17:01:58 <mtreinish> #topic Cancelling meetings for the rest of the year (mtreinish)
17:02:37 <mtreinish> so I was planning on cancelling the meeting for the next 2 weeks
17:02:52 <giulivo> indeed was asking the same thing, so we'd skip 2nd of jan too
17:02:53 <mtreinish> so we'd pick up the meeting again starting the second week of Jan.
17:02:58 <afazekas> ok
17:03:02 <dkranz> ok
17:03:20 <mtreinish> giulivo: yeah I'll be working the 2nd but it's a vacation time
17:03:30 <mtreinish> so I think waiting a week is probably best
17:03:42 <giulivo> yep same is from me too
17:03:56 <mtreinish> #info next meeting is Jan. 9th at 2200UTC
17:04:22 <mtreinish> ok that's nice and quick let's move on to the next topic
17:04:31 <mtreinish> #topic Blueprints
17:04:36 <mtreinish> giulivo: this is you
17:04:43 <mtreinish> you've got a bunch there on the agenda
17:04:56 <giulivo> yeah I think I moved most in an usable state
17:05:08 <giulivo> please triple check if you like but there are some which I'm asking for help
17:05:30 <sdague> giulivo: got a link of a particular view you like?
17:05:47 <giulivo> https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests is one, is it okay to approve or shall we just run the tests authenticating using latest keystone api?
17:06:25 <mtreinish> giulivo: we want to use all api versions that get registered.
17:06:35 <mtreinish> which is why we have both v1 and v2 for cinder and glance
17:06:51 <giulivo> oh but that bp isn't about the tests, it is about how tempest should auth
17:06:52 <mtreinish> until v2 is removed from keystone we'll need to test both
17:07:12 <mtreinish> giulivo: it's the same thing
17:07:17 <andreaf> giulivo: indeed, things like tokens and tenant isolation
17:07:25 <mtreinish> because it's still testing
17:07:30 <sdague> mtreinish: so how do you decide what yuo use for tenant isolation?
17:07:31 <andreaf> atm tenant isolation is v2 only
17:07:38 <sdague> because you do need to pick on that
17:07:48 <mtreinish> sdague: yeah it's only v2 right now
17:07:53 <andreaf> sdague: I'd like to have that as config
17:08:08 <mtreinish> andreaf: +1 yeah we should make it configurable
17:08:18 <giulivo> I'd +1 the config option too
17:08:22 <andreaf> we can decide whether to have two runs or focus on the latest stable version
17:08:43 <giulivo> so I'd approve and add that into the whiteboard, makes sense?
17:09:06 <andreaf> I didn't start working on that bp yet, it will be next year
17:09:07 <sdague> yep
17:09:13 <sdague> giulivo: +1 for approval
17:09:22 <mtreinish> giulivo: yeah that's fine
17:09:23 <giulivo> one more https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client , there is some actual work done
17:09:41 <mtreinish> giulivo: I'd argue that, what's been pushed out just adds some extra methods
17:09:50 <mtreinish> but nothing is actually used
17:10:38 <mtreinish> I've looked at this before and things are very different between xml and json that doing it that way doesn't save that much code
17:11:03 <giulivo> so this is eventually obsoleted
17:11:06 <mtreinish> if they want to work through it I'm not opposed to it. But, it feels like effort could be better spent on something else
17:11:28 <mtreinish> and there won't be any great code deduplication in the end
17:12:02 <giulivo> ok next is this https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone , do we at all want components to be testable like this?
17:12:02 <43UAACPJ0> mtreinish: We should make sure they understand what you do
17:12:43 <mtreinish> 43UAACPJ0? that's you dkranz
17:12:44 <dkranz> mtreinish: That strange number was me
17:13:03 <dkranz> mtreinish: I have been having strange irc problems and am using the web client
17:14:20 <mtreinish> giulivo: I think that one is fine, Swift is somewhat more common by itself. I'm not sure it makes sense for the other projects though
17:14:37 <afazekas> giulivo: if we would like , probably we will need an additional tempauth job on keystone and tempest  and on devstack
17:14:47 <notmyname> mtreinish: ?
17:15:06 <afazekas> if we would like , probably we will need an additional tempauth job on swift and tempest  and on devstack
17:15:12 <mtreinish> notmyname: swift testing in tempest without keystone
17:15:18 <mtreinish> there is a bp up for review on that
17:15:58 <mtreinish> afazekas: I think the bigger question is does that fit in tempest
17:16:00 <notmyname> let me know if you have any questions on that from the swift side of things
17:16:03 <sdague> so I'm pretty mixed on that at this point, because the review is a ton of ifdefing
17:16:11 <mtreinish> because tempest is about integration testing
17:16:23 <mtreinish> sdague: ok, I haven't looked at the review
17:16:34 <mtreinish> notmyname: thanks, will do
17:16:41 <sdague> and, as mtreinish says, tempest is about integration testing.
17:16:46 <giulivo> FWIW, I'm more on the side of a "no that doesn't go into tempest" except as thirdparty maybe
17:17:09 <dkranz> sdague: If they really want to do this maybe a fake keystone would be better than ifdefs?
17:17:31 <giulivo> I think it's probably more appropriate to put standalone tests in the swift project
17:17:39 <sdague> giulivo: right, agreed
17:17:53 <mtreinish> giulivo: yeah that makes sense
17:18:04 <sdague> if the tests don't need any other open stack services, it's not really clear to me why it would be in tempest vs. just in a swift project
17:18:26 <dkranz> sdague: There is still the goal of api stability by tempest
17:18:44 <afazekas> sdague: double checked API stability ?
17:18:46 <dkranz> sdague: It's not just about integration
17:19:23 <notmyname> sdague: because a test for an openstack project would be in tempest? how is "because this functionality doesn't cut cross projects, remove it from tempest" a thing?
17:19:46 <dkranz> notmyname: It's not
17:19:55 <notmyname> good :-)
17:20:21 <notmyname> that not what I understood from sdague's comment
17:20:21 <sdague> notmyname: it's more about the extra overhead of maintaining a mode of "I don't want to use other openstack services"
17:20:27 <dstanek> this is very similar to our goal of moving some of the keystoneclient tests into tempest
17:20:47 <dstanek> making sure keystone in master doesn't break old clients
17:21:03 <notmyname> sdague: I think it's more about isolating test failures to the lowest set of variables
17:21:15 <dkranz> notmyname: What sdague just said is the real root of the matter. swift is the only project that does not use keystone as a matter of course
17:21:55 <giulivo> dstanek, there is a blueprint for that though, as that is a problem shared with clients of other components
17:22:02 <notmyname> dkranz: keystone is a supported, official, upstream option for auth. it doesn't use or not use keystone more than any other option
17:22:26 <sdague> notmyname: so then why does it matter if we test with keystone?
17:23:06 <sdague> is this really a gate question?
17:23:50 <dkranz> notmyname: Doesn't swift already have a job with functional tests that are not in tempest but are "real"?
17:23:55 <notmyname> I've not been part of this discussion to date. it seems the advantage of using tempauth is to have less dependencies for testing and thus isolate failures. but testing with keystone is vital and should never be removed. isn't the question on using non-keystone tests?
17:24:14 <notmyname> dkranz: yes, we have a set of functional tests in the codebase
17:24:26 <dkranz> notmyname: So why not put this case there?
17:25:27 <notmyname> dkranz: auth integration is a pluggable mechanisms with a standard API. it shouldn't matter which you use. but FWIW, we aren't testing keystone in our functional tests
17:25:51 <dkranz> notmyname: At the end of the day, all of tempest is really more like a smoke test. It is never going to test everything or every deployment configuration.
17:26:13 <dkranz> We are already testing swift with keystone.
17:26:26 <dkranz> Perhaps we should take this to the ml?
17:26:35 <sdague> so honestly, right now, I'd rather not take the complexity here to make tempest function without auth, especially as we have tons of things we need in the cycle.
17:26:48 <sdague> sure
17:27:10 <notmyname> so this is a new discussion to me. I'm not sure really what the arguments are, if any, or even what the options are
17:27:25 <sdague> notmyname: first I saw of it was yesterday when I was reviewing code :)
17:27:29 <sdague> so it's pretty new to me as well
17:27:31 <mtreinish> notmyname: this is the first time I've seen it too
17:27:45 <giulivo> let's continue after the meeting
17:27:49 <giulivo> I've a couple more bps
17:27:50 <mtreinish> ok, giulivo are they any other blueprints
17:27:55 <mtreinish> otherwise we'll move on
17:28:05 <mtreinish> giulivo: ok
17:28:16 <giulivo> this https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api isn't really clear in scope to me
17:28:36 <mtreinish> giulivo: there was a summit session about that
17:28:45 <giulivo> auch, I missed it indeed :(
17:28:47 <giulivo> approved
17:29:01 <mtreinish> no I don't think it was
17:29:07 <sdague> is bknudson about
17:29:13 <sdague> he and I were talking about this one the other day
17:30:03 <giulivo> and what is the correct status for this then? approved but discussing?
17:30:04 <sdague> giulivo: so I think on something like this, can you flag that we need someone to show at the meeting to talk about it
17:30:22 <sdague> giulivo: honestly, I don't think approved in current state
17:30:33 <giulivo> make sense, last one
17:30:37 <sdague> it needs some cleanup based on follow up conversations to make sure we got to the same page
17:30:51 <dstanek> giulivo: i think there are two things i've heard in regards to client testing
17:31:28 <dstanek> one is that we want to make sure that the api of the client doesn't break (not just the rest api of the server)
17:31:28 <giulivo> https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors I think we need to figure if it's worth closing it and in that case how to track "failures" which should be fixed by devs, which was a valid concern from dkranz
17:32:00 <dstanek> the other is that we currently have a set of tests in keystone that get run against different client versions
17:32:19 <dstanek> the test actually checks out older releases of the client to run the tests
17:32:22 <sdague> dstanek: so we've been over that one on the ML already as well. The reason keystone had a break here was the fact that they didn't test the rest api in question
17:32:26 <dkranz> giulivo: There is no reason for this to stay open except for possible "pr" value
17:32:46 <sdague> dstanek: right, and that's a different discussion about job setup
17:33:44 <sdague> giulivo: I'd close https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors as done
17:33:57 <mtreinish> giulivo: ok, is that it for blueprints then?
17:34:03 <andreaf> https://blueprints.launchpad.net/tempest/+spec/input-scenarios-for-scenario
17:34:21 <mtreinish> andreaf: that's approved
17:34:55 <mtreinish> so let's move on since we spent a lot of time on this topic
17:35:01 <mtreinish> #topic Bugs
17:35:06 <giulivo> yeah sorry about that but I thought was worth it
17:35:16 <mtreinish> sdague: this has your name on it
17:35:18 <sdague> yep
17:35:42 <sdague> so we managed to get down to 73 bugs. Looks like we are back up to 89 now.
17:36:12 <sdague> most of those new bugs aren't tempest bugs, per usual, people just dumping a random stacktrace and calling it a tempest bug
17:36:46 <sdague> so the policy we ended up taking was marking those invalid for tempest, and if there was moderate debug in it, putting it to the right project
17:37:00 <sdague> I still owe writing up our policy there... it's on my list
17:37:15 <mtreinish> sdague: ok, that's what I normally do when I come across those bugs
17:37:17 <sdague> but I'd also like to see if we can stay on top of new bugs and get them cleared out
17:37:24 <adalbas> sdague, i was gathering a few thoughts on that since the bug triage too
17:37:31 <adalbas> i ll send them over to you
17:37:47 <andreaf> sorry I lost connectivity to my workstation
17:37:49 <dkranz> sdague: https://review.openstack.org/#/c/61850/ was just approved which should help people diagnose
17:38:16 <dkranz> and figure out which project is at fault
17:38:24 <sdague> cool
17:38:46 <dkranz> sdague: I will post to the ml about it
17:38:55 <sdague> starting in the new year I'd also like to do a review of all Critical and High bugs in this meeting as a standing issue
17:39:01 <sdague> we should be working to actually close those out
17:39:06 <mtreinish> sdague: +1 that's a good idea
17:39:06 <sdague> that's about it from me on that
17:39:20 <adalbas> +1
17:39:50 <mtreinish> sdague: ok
17:39:55 <mtreinish> then let's go to the next topic
17:39:58 <andreaf> mtreinish: about bp above, I just wanted to say reviews are welcome :)
17:40:07 <mtreinish> #topic Neutron testing
17:40:28 <mtreinish> so mlavalle said he can't make it
17:40:49 <mtreinish> but he said that his api gap analysis is making progress and he's almost done
17:41:03 <mtreinish> and that there have been volunteers who have started working on the testing to fill the gaps
17:41:36 <mtreinish> I haven't been watching this too closely the past week
17:41:38 <sdague> anyone else planning to go to montreal besides mtreinish and I?
17:42:17 <afazekas> What kind of additional 'debug'  should be done at SSHTimout exception in order  to solve this kind of issues sooner  https://bugs.launchpad.net/tempest/+bug/1253896 ?
17:42:19 <mtreinish> sdague: I guess it'll just be us
17:42:20 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress]
17:43:19 <sdague> afazekas: I think you need to ask the neutron folks on that. I know they've solved 5 races so far working on that bug
17:43:40 <salv-orlando> We have patches in review that *at least in my env* improve a lot the situation. Those patches are the same that are being worked on for improving parallel testing
17:43:46 <afazekas> We could dump the ovsdb , we could try to make an ssh connection with openssh instead of paramkio,  we can try to make connection inside the ns, we can print the vm console
17:44:31 <salv-orlando> also, we have another patch from marun that solves another root cause of bug 1253896 pertaining a problem with notification to the dhcp agebt
17:44:31 <mtreinish> salv-orlando: ok cool
17:44:33 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] https://launchpad.net/bugs/1253896
17:44:34 <afazekas> Attach any type of debuger to anything :)
17:44:49 <andreaf> ssh connection is really testing key injection
17:45:14 <andreaf> to check the network pluming is in place ping should be good enough
17:45:15 <salv-orlando> actually just ovs-vsctl list Interface and ovs-vsctl list Port would do a lot
17:45:37 <afazekas> we can try to make second attempt with the built in cirros password, to make sure not just the key missing
17:46:01 <marun> andreaf: disagree
17:46:05 <salv-orlando> andreaf: the thing is that there are several places in which the plumbing might not have worked that a ping is not enough to perform a diagnosis.
17:46:10 <marun> no guarantee we are pinging the right thing
17:46:30 <afazekas> marun: agreed
17:46:40 <dkranz> marun: Indeed. I had this issue in practice, resolving to the wrong machine
17:46:42 <sdague> right neutron is clever enough in ports, that we need to actually know we are talking to the right guest
17:46:44 <salv-orlando> namely: floating IP wiring, router interface wiring, VIF port wiring, DHCP port wiring, dnsmasq configuration, timing of DHCPDISCOVER messages, and not last, security filters
17:47:33 <sdague> salv-orlando: so from a tempest perspecitive ovs commands aren't really things we can/should be driving
17:47:49 <salv-orlando> sdague: I think either I or marun can do that
17:47:53 <sdague> if there was a neutron debug api that did that and gave us results, that would be cool
17:48:05 <sdague> tempest really needs to only hit REST apis
17:48:29 <marun> sdague: not at the top of the list, but usable health-checks available via rest is on it
17:48:30 <salv-orlando> sdague: I was thinking of adding this information to the dump of low-level information tempest already does (namespace and iptables)
17:48:32 <andreaf> marun: salv-orlando: I'm not saying we should skip ssh completely, but we could split into ping first and ssh than, or telnet to port 22 and ssh then
17:48:45 <marun> andreaf: we already do that
17:48:46 <andreaf> to help debugging issues
17:48:51 <sdague> salv-orlando: yeh, so that's sort of an ugly work around that we really want to be removing
17:48:53 <salv-orlando> andreaf: test_network_basic_ops already works this way
17:48:57 <sdague> because it only works in the single node case
17:49:08 <afazekas> sdague: On failure we can and should do more on the gate, but the normal test run should not be broken, just because some internal thing is changed
17:49:16 <andreaf> marun: ok sorry I've not followed closely enough
17:49:41 <salv-orlando> sdague: np, I also am working on having tempest report the operational status for the various bits which might impact a failure. This will tell us at least what's not wired. I just wish we had a way to gather whether the VM received the IP or not.
17:49:52 <marun> andreaf: it's been there since may iirc.  previously we just pinged, but it became apparent that it didn't guarantee anything so ping, then ssh has been the rule since.
17:50:11 <sdague> salv-orlando: can we get it out of the nova-console?
17:50:35 <mtreinish> salv-orlando: yeah that should be on the console log
17:50:41 <afazekas> salv-orlando: we can ask the vm to run an user-data script, and print info into the serial console
17:50:51 <salv-orlando> sdague: yeah, I forgot we can access the text consol
17:50:52 <salv-orlando> e
17:50:59 <marun> salv-orlando: is that capability something that could be added to cloud-init?
17:51:02 <sdague> but the vm should already dump that stuff in the console
17:51:15 <marun> sdague: it does
17:51:36 <marun> sdague: at least for a cirros image we should be able to parse console.log
17:51:36 <salv-orlando> ok, so this is sorted. We just need to access the console log.
17:51:47 <mtreinish> ok, I ahte to cut things short, here but we're running out of time
17:51:48 <sdague> right, there's an API for that
17:51:54 <mtreinish> so we can pick this up after the meeting
17:51:56 <sdague> yep
17:52:05 <mtreinish> #topic On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
17:52:10 <dkranz> sdague: One last thing can we get a push on the network scenarios of Yair Fried?
17:52:27 <sdague> dkranz: sure, I went through a bunch of them yesterday
17:52:31 <mtreinish> ok giulivo in 8 min :)
17:52:42 <mtreinish> dkranz: yeah I'll try to take a look at them soon too
17:52:42 <dkranz> sdague: Me too, but there are a lot, which is good
17:52:49 <dkranz> next
17:53:06 <giulivo> I'm not sure this is quick, but basically, I noticed heat people did a pretty good job at creating blueprints for different test cases
17:53:21 <giulivo> and configure dependencies accordingly as they use blueprints to track progresses and assignments
17:53:28 <mtreinish> giulivo: we can save it for the next meeting if you'd prefer
17:53:33 <giulivo> now, that is different from random blueprints asking for new tests
17:53:34 <bknudson> sdague: what's up?
17:53:48 <giulivo> mtreinish, let me finish so people can start making up an idea
17:53:50 <dkranz> mtreinish: We need to resolve this issue now because there is going to be an incoming of heat tests
17:53:55 <giulivo> and maybe rediscuss next week
17:54:01 <mtreinish> giulivo: ok
17:54:05 <sdague> giulivo: well, next week is 3 weeks :)
17:54:06 <dkranz> next two meetings were cancelled
17:54:12 <giulivo> so my intermediate proposal is "one blueprint, per project, per release"
17:54:29 <giulivo> but that could make everyone unhappy, so I can't bet on it
17:54:31 <sdague> giulivo: +1 on that for now
17:54:38 <mtreinish> giulivo: yeah that's fine
17:54:51 <mtreinish> the blueprint can be divided with work items and in the description
17:54:52 <dkranz> +1
17:54:55 <sdague> until we get storyboard, I think the artifact overhead is too much on lp
17:55:12 <giulivo> and if they're willing to manage ?
17:55:17 <giulivo> (their own?)
17:55:25 <sdague> giulivo: we need to manage them as well
17:55:33 <sdague> if they are tempest blueprints
17:55:37 <dkranz> sdague: Using bugs withing their project as another idea
17:56:05 <dkranz> sdague: I think we should say that any mechanism other than multiple blueprints in tempest, or using tempest bugs, is ok
17:56:06 <sdague> yeh, honestly, I just don't want to invest much in this system, because I hate launchpad so much :)
17:57:11 <mtreinish> giulivo: well that was pretty quick :)
17:57:17 <sdague> last thing then
17:57:21 <giulivo> lucky you
17:57:23 <mtreinish> yep
17:57:26 <giulivo> :)
17:57:27 <sdague> holidays are coming up, and I know folks will be out
17:57:28 <mtreinish> #topic Critical Reviews
17:57:46 <sdague> so if people could take an extra hour or two before leaving and look at the queue
17:57:54 <sdague> it's grown a bit in the last couple of weeks
17:58:03 <sdague> and a lot of it is pretty straight forward stuff to get in
17:58:38 <sdague> we're currently at 122 open reviews
17:58:38 <mtreinish> sdague: yeah I plan to do that today (my last day working)
17:59:18 <mtreinish> well with a min left I don't think there's really enough time to talk about any specific reviews
17:59:24 <mtreinish> so let's just end here
17:59:34 <mtreinish> thanks everyone
17:59:48 <mtreinish> #endmeeting