17:01:55 <mtreinish> #startmeeting qa
17:01:55 <openstack> Meeting started Thu Dec  3 17:01:55 2015 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:59 <openstack> The meeting name has been set to 'qa'
17:02:10 <mtreinish> hi, who's here today?
17:02:14 <dwalleck> o/
17:02:17 <ylobankov> hi
17:02:27 <jswarren> hello
17:02:50 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_December_3rd_2015_.281700_UTC.29
17:02:55 <mtreinish> ^^^ Today's agenda
17:04:09 <mtreinish> andreaf_, dtroyer, sdague, mkoderer, afazekas, jordanP: around?
17:04:43 <sdague> yeh, sort of. If any of my stuff can be front loaded that would be great
17:04:44 <_dpaterson> o/
17:05:07 <hogepodge> o/
17:05:13 <mtreinish> lets get started
17:05:14 <jordanP> yep
17:05:22 <mtreinish> #topic Spec Reviews
17:05:28 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:05:36 <mtreinish> we've got a few open spec reviews
17:05:41 <mtreinish> it'd be good to get eyes on them
17:05:45 <andreaf> o/
17:05:53 <mtreinish> I know SpamapS has been bugging me about his counters one lately :)
17:06:34 <mtreinish> does anyone have a specific spec they'd like to discuss?
17:06:41 <andreaf> I had one questions regarding the resource spec
17:06:47 <dwalleck> mtreinish: I talked with lifeless about the run one before I left on vacation and he refreshed me about what we discussed in Tokyo. That spec should be up in the next few days
17:06:51 <andreaf> #link https://review.openstack.org/#/c/173334/6/specs/tempest/tempest-resources.rst,cm
17:07:06 <mtreinish> dwalleck: ok, cool
17:07:07 <mtreinish> andreaf_: sure
17:07:58 <andreaf> mtreinish: the spec introduces the concept of filters that can be applied to test/test-classes to express dependency on test resources
17:08:21 <andreaf> the question is what the behaviour should be in case such resources are not available, skip or fail?
17:08:56 <mtreinish> hmm, maybe doing a 3 condition kinda thing: hard-requires(fail if missing), soft requires(skip if missing), and preferred
17:09:04 <andreaf> for instance if no image is configured - currently we fail if image_ref is not set and a test that requires it is executed
17:09:21 <_dpaterson> +1 on three condition
17:10:04 <jordanP> -1 too much complexity
17:10:07 <jordanP> :)
17:10:09 <mtreinish> haha
17:10:10 <andreaf> mtreinish: ok, that sounds like a good idea, but we need to provide some guidance about how to use that
17:10:37 <mtreinish> andreaf: yeah, we should document it very clearly
17:10:48 <jordanP> we should just fail, be vocal about it
17:11:14 <mtreinish> that's something we need to work on in general is having a better guide for test writing. There is a lot of complexity already in the internal machinery in tempest
17:11:20 <mtreinish> but that's a seperate topic
17:11:25 <jordanP> exactly !
17:11:46 <jordanP> landing this auth credentials work is an example
17:11:50 <mtreinish> jordanP: so your just saying requires (fail) and prefered
17:11:52 <jordanP> it was not an easy road
17:11:56 <jordanP> yes
17:12:29 <jordanP> and jlanoux also had a hard time with his ssh validation work
17:12:35 <andreaf> in a sense skipping by missing resources is kind of implicit - so perhaps we should just fail and not try to be nice - if someone doesn't want to run a test she/he should skip it otherwise?
17:13:25 <jordanP> and think this was our policy as of now 'fail hard in case of misconfiguration'
17:13:31 <mtreinish> I think that makes sense
17:13:34 <mtreinish> jordanP: yeah it is
17:13:41 <mtreinish> it's part of being explicit
17:13:43 <andreaf> for instance @prefered(small-image) means that if a small-image is not found it will go with any image, and if none is found it will fail
17:14:08 <jordanP> that is going to require "a lot" of logic
17:15:16 <andreaf> jordanP: not sure that's going to be that complex, but I would start implementing requires as a first step
17:15:38 <andreaf> mtreinish, jordanP: in any case I think it's settled for failure rather than skip then?
17:15:45 <mtreinish> yeah I think so
17:15:58 <jordanP> I think that's the easiest way and a good start too
17:15:58 <andreaf> ok. Thanks
17:16:09 <mtreinish> ok is there anything else on specs?
17:16:14 <dwalleck> I had one other question about the resources spec. It introduces the tools section to handle the issue of different commands being necessary for different distros, but how is the problem of handling parsing of the different commands? The remote client still expects the output of the command to be in the format of the original one
17:16:34 <andreaf> dwalleck: heh, thanks I was going to bring that on as well
17:16:42 <dwalleck> It's a hot button issue in DefCore right now
17:16:51 <jordanP> where "ip" is in /sbin or /usr/sbin , the output shouldn't change right ?
17:16:58 <jordanP> *whether
17:17:11 <mtreinish> dwalleck: I think that's a separate discussion, and honestly the defcore concerns are secondary to the work in the spec
17:17:26 <mtreinish> jordanP: it's more if what if you wanted to use a freebsd or solaris image
17:17:32 <mtreinish> the output and the path would be different
17:17:49 <andreaf> dwalleck: so the alternative would be to define what we want to obtain rather than the specific tool, so if you want to get an IP use this helper
17:17:49 <dwalleck> mtreinish: That's fair. Being able to use different images is really my concern
17:18:00 <jordanP> I see, now I remember this recent thread on the ML...
17:18:05 <andreaf> dwalleck: but then you need to maintain such helpers
17:18:23 <andreaf> guest image specific ways to extract certain data from your OS
17:18:35 <andreaf> and tempest is not the right place for that
17:18:41 <mtreinish> dwalleck: realistically we're talking about a very small subset of tests too
17:18:45 <dwalleck> andreaf: Right. I have a few ideas on how this could be implemented cleanly
17:18:55 <andreaf> so you need a plugin mechanism for that or so
17:19:14 <dwalleck> andreaf: That's exactly what I was thinking
17:19:33 <andreaf> dwalleck: that sounds like a separate spec though
17:19:49 <andreaf> I think having the path to the tool can be a first approximation
17:20:20 <dwalleck> mtreinish: I know they're a small set now, but they're extremely important for me and I see there being a lot more
17:20:36 <dwalleck> But that's outside the scope of this conversation. Thanks for humoring me :-)
17:21:03 <mtreinish> ok, lets move then
17:21:12 <mtreinish> #topic Priority Items
17:21:19 <mtreinish> #link https://etherpad.openstack.org/p/mitaka-qa-priorities
17:21:35 <mtreinish> so today being M-1 (well more or less)
17:21:50 <mtreinish> I think we should go over the open M1 items on the etherpad just to see where they stand
17:22:04 <mtreinish> I see 3 on the etherpad
17:22:18 <mtreinish> the first is: Drop extras.d from devstack
17:22:24 <mtreinish> sdague: ^^^ any update on that?
17:22:31 <sdague> mtreinish: yep, just updating etherpad - https://review.openstack.org/#/c/252944/
17:22:37 <sdague> I posted the review for that this morning
17:22:43 <mtreinish> #link https://review.openstack.org/#/c/252944/
17:22:45 <mtreinish> sdague: ok, cool
17:22:49 <sdague> I feel like I did enough warning we should be good to go on it
17:23:03 <sdague> though I think I got hit by gerrit restart and lost having tests run on it
17:23:25 <mtreinish> heh
17:23:45 <sdague> anyway, rechecked now, we should merge it tomorrow
17:23:46 <mtreinish> sdague: yeah I agree, you've been warning about that and posting to the ml for long enough
17:23:49 <mtreinish> ok cool
17:24:09 <mtreinish> the next on the list is: Tempest Microversion support
17:24:19 <mtreinish> which is owned by gmann and oomichi
17:24:29 <mtreinish> but since they're in JST and probably asleep
17:24:41 <mtreinish> they've got reviews up for starting most of that here:
17:24:44 <mtreinish> #link https://review.openstack.org/#/q/status:open++branch:master+topic:bp/api-microversions-testing-support,n,z
17:25:00 <mtreinish> prioritizing review on those would be great
17:25:43 <mtreinish> I don't think the branch is quite enough to close the bp as the spec is written, but it's mostly there
17:26:01 <sdague> yeh, there was a sticking point about how to put the config in from devstack
17:26:23 <sdague> because we should really bypass devstack for this I think
17:26:29 <sdague> and go from d-g => tempest
17:27:30 <mtreinish> sdague: I think that's probably fine. It's not like devstack knows about the microversions being deployed
17:27:38 <sdague> although...
17:27:47 <sdague> now that I think about it, devstack kind of does know
17:27:55 <sdague> because it pulled master
17:28:20 <sdague> I have to rethink this one, I'll propose something
17:28:26 <mtreinish> sdague: ok :)
17:28:53 <mtreinish> anything else to discuss on microversion testing?
17:29:03 <andreaf> sdague, mtreinish: still there is a difference between the available microversions and what we want to test in the gate, so we probably need to drive at least some of it via d-g anyways
17:29:23 <sdague> andreaf: yes, for sure
17:30:37 <mtreinish> ok, the last thing open thing on the list for M1 is: OpenStack Health group-by / filter-by metadata key
17:30:43 <mtreinish> which andreaf is listed as owning
17:30:48 <mtreinish> andreaf: any update on that?
17:31:15 <andreaf> mtreinish: for group-by there is a patch up, let me dig the link out
17:31:38 <andreaf> but filter-by is not started yet
17:31:57 <andreaf> and unfortunately some of the folks helping on this are not around anymore
17:32:08 <mtreinish> andreaf: refresh my memory what does the filter-by entail?
17:32:09 <andreaf> so it may slow down
17:32:36 <andreaf> filtering the graphs on the page by a value on a metadata key
17:33:01 <andreaf> e.g. show all the runs where build_pipeline = periodic
17:33:07 <andreaf> or something like that
17:33:17 <mtreinish> oh, don't we already have that?
17:33:24 <mtreinish> there is the search bar which dynamically filters
17:33:52 <andreaf> I though we could search by name only?
17:34:18 <mtreinish> oh, this is a secondary key value pair, I get it now
17:34:31 <andreaf> mtreinish: group-by #link https://review.openstack.org/#/c/240453/
17:34:48 <mtreinish> ok, cool thanks
17:35:16 <mtreinish> before we move on, does anyone want to discuss another priority item? (one that's not necessarily M1 targetted)
17:36:16 <mtreinish> ok, let's move on
17:36:33 <mtreinish> #topic Eslint-config-openstack review approval policy (krotscheck)
17:36:48 <mtreinish> krotscheck: around? this is your topic
17:38:51 <mtreinish> ok, I guess he's not around
17:38:55 <mtreinish> let's move on
17:39:06 <mtreinish> #topic Tempest
17:39:22 <mtreinish> does anyone have anything to discuss on tempest this week?
17:39:50 <_dpaterson> mtreinish: config script
17:40:12 <mtreinish> dwalleck: what about it? :)
17:40:16 <_dpaterson> Daniel from posted something on it on the ML
17:40:24 <dwalleck> huh?
17:40:32 <_dpaterson> He meant dpaterson :)
17:40:44 <mtreinish> oh, yeah I've been meaning to chime in on that thread
17:40:49 <mtreinish> dwalleck: sry, tab complete fail
17:41:00 <_dpaterson> I guess I still understand the friction on getting it upstream
17:41:07 <_dpaterson> don't understand rather
17:41:24 <_dpaterson> blueprint is here: https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
17:42:12 <mtreinish> _dpaterson: that spec has been abandonded: https://review.openstack.org/#/c/94473/
17:42:34 <mtreinish> which was the issue originally, is there was discussion about the scope of the tool and where we should cut off what it does and what it doesnt
17:42:44 <mtreinish> and then interest just died down and nothing happened
17:43:23 <_dpaterson> Okay, so I'd like to assist Daniel with re-addressing it, do we need to create a new spec?
17:43:36 <andreaf> _dpaterson, mtreinish: I think the tool as it is does not fit in tempest, as some parts of it are already in
17:43:49 <mtreinish> dwalleck: yeah I think starting a fresh spec is the right place to start
17:43:51 <jordanP> what part of it ?
17:44:00 <_dpaterson> andreaf: where?
17:44:40 <andreaf> for instance the discovery part: that's already in the verify-config-tool
17:44:55 <andreaf> and that's shouldn't be used for anything else than verification
17:45:30 <andreaf> the resource provisioning part is something that is not in yet, but that we plan to implement as part of the resource bp
17:45:33 <jordanP> the point of the tool is to help generate tempest.conf, some sort of discovery is required
17:45:51 <_dpaterson> andreaf: creating the config and verifying it are two different things, where there is overlap refactoring to common code for discovery would be done
17:46:04 <jordanP> +1
17:46:24 <andreaf> passing fragments of configs in is something as trivial as append them to the bottom of the file but still is something we could add to a tempest run
17:46:39 <mtreinish> jordanP, _dpaterson: the verify tool, can make a new config fwiw. You just need to give it the bare min to do the discovery
17:46:47 <andreaf> jordanP, _dpaterson: sure I understand that, but you don't want to use discovery to generate a test config file
17:47:18 <jordanP> andreaf, I don't understand why not
17:47:21 <_dpaterson> andreaf:  maybe we should take a step back.
17:47:35 <_dpaterson> A configuration generation tool would be useful
17:47:52 <andreaf> because if you generate your configuration by discovery you may miss things
17:48:09 <_dpaterson> Implementing it can be done a bunch of different ways, lets get a new spec going and move the discusion there.
17:48:11 <andreaf> let's say that a feature is disabled by mistake - you will stop testing it and never know about it
17:49:17 <_dpaterson> andreaf: the tool does not dismiss the user from vefifying his environment is correct.  It is a jump start.
17:49:42 <mtreinish> in the interest of time, I think we should take this up in the -qa channel post meeting
17:49:55 <mtreinish> because we're at ~10mins left
17:49:58 <jordanP> and it does not prevent running verify-tempest afteward
17:50:14 <_dpaterson> I'll work with Daniel on new spec and take discussion to #qa
17:50:23 <mtreinish> _dpaterson: ok, thanks
17:50:34 <mtreinish> let's move on
17:50:55 <mtreinish> #topic Critical Reviews
17:51:10 <mtreinish> Does anyone have any reviews they'd like to get extra eyes on?
17:52:39 <mtreinish> one I wanted to raise up was:
17:52:41 <mtreinish> #link https://review.openstack.org/216303
17:53:01 <mtreinish> I'm not really sure how to weigh in on it. I don't have a lot of experience with that kinda of testing
17:53:17 <mtreinish> it also raises a question about having frameworks from outside tempest being added to tempest-lib
17:53:26 <mtreinish> but that's something I've generally been a fan of :)
17:54:03 <jordanP> I am not sure why tempest-lib should host an HTTP fuzzer
17:54:37 <jordanP> :)
17:54:50 <mtreinish> heh, fair enough
17:54:53 <jordanP> I am sorry I tend to first say no and then maybe
17:55:17 <mtreinish> the review was just sitting there, so I just wanted people to take a look and talk about it
17:55:34 <mtreinish> it's not my patch, so I'm not really invested in it :)
17:55:38 <jordanP> and also this looks like a preliminary work and we don"t see the practical use yet with this patch
17:56:16 <jordanP> hum I missed the dependent patch
17:56:38 <jordanP> I'll have to review it more carefully
17:57:09 <jordanP> but it really looks like an HTTP fuzzer and there's ton of this already available
17:58:03 <mtreinish> jordanP: ok, that's good feedback to have on the review
17:58:11 <jordanP> yep I'll leave a comment
17:58:18 <mtreinish> ok are there any other reviews this week?
17:59:27 <mtreinish> ok, well I guess lets end here then
17:59:30 <mtreinish> thanks everyone
17:59:32 <mtreinish> #endmeeting