17:00:04 <gema> #startmeeting Tailgate
17:00:05 <openstack> Meeting started Thu Jul  2 17:00:04 2015 UTC and is due to finish in 60 minutes.  The chair is gema. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:09 <openstack> The meeting name has been set to 'tailgate'
17:00:29 <gema> hi everyone, can anyone here for today's meeting say something, plz :)
17:00:32 <gema> o/
17:01:02 <jasonsb> morning gema
17:01:09 <gema> evening :D
17:01:23 <jasonsb> your a trooper
17:01:36 <gema> exciting times :D
17:01:37 <jasonsb> thank you for setting up the meeting bot
17:01:52 <gema> you are welcome, it was fun
17:02:18 <gema> shall we wait a couple of mins to give people time to show up if they are going to?
17:02:41 <jasonsb> certainly.  i was going to ask about hogepodge
17:02:50 <gema> he said he'd be here
17:03:07 <jasonsb> i didn't fully grasp your conversation from few days back
17:03:11 <hogepodge> hi
17:03:17 <hogepodge> o/
17:03:18 <gema> hi hogepodge
17:03:26 <gema> clee: you around?
17:03:27 <jasonsb> hogepodge: greetings
17:03:48 <localloop127> i'm here. was not paying attention when things started
17:03:59 <gema> goodio, let's get started then
17:04:11 <gema> I have a sort of agenda
17:04:21 <gema> #topic Progress on last week's actions
17:04:48 <gema> I did the ones assigned to me, bot, sending email to the mailing about the meeting and adding kloudbuster to the test doc
17:04:56 <gema> #link test strategy doc: https://docs.google.com/document/d/19N4lZJ5mYkQbkXrtBGntf7C1bWvwt1RKoSU8NerAphQ/edit#
17:05:03 <gema> spyderdyne is not around
17:05:09 <gema> so we'll have to postopone asking him about rally
17:05:23 <clee> gema: kind of!
17:05:30 <gema> clee: go ahead :D
17:05:30 <clee> gema: I'm in a dreamhost meeting
17:05:42 <gema> ah, ok, so half here
17:05:45 <clee> gema: I emailed dkranz but have yet to receive a reply
17:05:54 <gema> let's start with the defcore topic next then
17:06:00 <gema> clee: we'll give you time to wrap up
17:06:08 <gema> #topic DefCore
17:06:21 <clee> gema: the DH meeting will probably last longer than this one, but I'll try to pay attention here when I can
17:06:29 <gema> ok
17:06:47 <gema> I had a conversation with hogepodge the other day and I will let him introduce himself and the defcore project
17:06:55 <gema> hogepodge: thanks for coming
17:07:16 <hogepodge> Hi,
17:07:55 <hogepodge> Defcore is a board-approved process that requires OpenStack clouds to pass a series of API tests as part of our trademark program.
17:08:37 <hogepodge> It originally came out of clause in the bylaws that stated that OpenStack branded products needed to pass a "faithful implementation of testing standards" (or some name like that), FITS
17:09:42 <hogepodge> We starting testing clouds against a test suite based on tempest a few months ago. They are non-admin API calls that check basic functionality of Nova, Swift, Cinder, and Keystone (and implicitly Glance and Neutron)
17:10:34 <hogepodge> We use a tool called Refstack to semi-anonymously collect test results (companies can voluntarily self-identify their results publicly for the Defcore process)
17:11:03 <hogepodge> The goal of defcore is to promote interoperability across public, hosted, and distributions.
17:11:24 <gema> hogepodge: could you define interoperability in this context, what do you mean by that?
17:12:03 <hogepodge> Interoperability would be a stable set of APIs that work consistently across clouds. So an application written for one cloud can work on a second cloud.
17:12:10 <gema> ok
17:12:44 <gema> and you also aim at keeping backwards compatibility in the future?
17:12:48 <gema> or is that out of scope?
17:12:51 <hogepodge> We still have a ways to go to get true interoperability, but the recently approved process is a good start, and our aim is to spend the next year really nailing down APIs and holding vendor's and devs accountable to making clouds work together.
17:13:17 <hogepodge> We try to have a lagging process, and are looking for APIs that will be around for a while or have a clear deprecation strategy
17:13:31 <gema> ack
17:13:59 <hogepodge> We also allow validation against the latest two approved standards to give vendors wiggle room.
17:15:03 <gema> jasonsb: do you still have questions?
17:15:26 <hogepodge> We meet weekly for general committee work, and also are starting up a new capabilities meeting to introduce new tests, with more attention on this cycle being given to images store and networking
17:15:52 <gema> hogepodge: sound like the capabilities meeting is a good one for us to attend also?
17:16:55 <hogepodge> We would love for as many members of this group to participate as possible. Defcore is meant to be a community driven process, and the tailgate team seems to have a number of overlapping concerns. The advantage to participating is you get the backing of the board and foundation in the approved test suite, and everyone who sells an OpenStack branded cloud or distro needs to meet that standard.
17:17:24 <gema> based on our conversation the other day, you said that tempest will keep maintaining any tests that defcore uses for certification, correct?
17:17:47 <gema> (instead of devolving them to the different projects)
17:18:08 <jasonsb> gema: good question
17:18:14 <hogepodge> gema yes, that's what the ptl of the qa team wants.
17:18:35 <hogepodge> gema we do have a procedure for introducing different test suites if necessary, though
17:18:46 <gema> hogepodge: do you have a link to that?
17:19:21 <hogepodge> gema in general, every document is located here: https://git.openstack.org/cgit/openstack/defcore/tree/
17:19:38 <gema> ack
17:19:44 <gema> #link https://git.openstack.org/cgit/openstack/defcore/tree/
17:19:54 <gema> #link http://eavesdrop.openstack.org/#DefCore_Committee_Meeting
17:20:28 <hogepodge> Our new schema includes a field for test repositories. https://git.openstack.org/cgit/openstack/defcore/tree/2015.next.json#n43
17:20:36 <gema> hogepodge: sounds like your test suites are the right place for a good number of the test cases we were aiming to be adding
17:20:38 <jasonsb> hogepodge: can you speak to the relationship of your testing vs big tent?
17:21:08 <jasonsb> looks like a take-it-slow approach, but i was wondering what the overall strategy is
17:21:17 <hogepodge> jasonsb: We're focusing on what would be controversially called the "core" right now. Compute, block storage, object storage, image repository, and networking.
17:22:03 <hogepodge> jasonsb: Nothing would preclude new OpenStack Powered programs from branching out to other projects. For example, a hypothetical "OpenStack Powered Database" program might have tests for Trove.
17:22:32 <hogepodge> jasonsb: but all changes to the projects in the program must be committee proposed and board approved
17:22:44 <jasonsb> ok, the core has a good home in tempest currently
17:23:59 <hogepodge> jasonsb: we want to be tied to tempest only for simplicity, but also are open to other test suites if we can't meet our goals with tempest only.
17:24:00 <gema> hogepodge: so if we wanted to get involved and start contributing , what would that look like?
17:24:33 <hogepodge> gema: the immediate spot we need help in right now is in determining capabilities for the upcoming proposed standards
17:24:55 <hogepodge> gema: we also need people to write non-admin api tests to help us fill the gaps in our testing, to add as capabilities
17:25:26 <gema> hogepodge: for someone like me, that is mainly a "black box tester" of clouds, what is a capability
17:25:29 <hogepodge> gema: If you have a cloud, we would also love for more people to run refstack-client against it and report on all of the passing tests (this is automated with a client-server model)
17:26:24 <gema> hogepodge: I can run your tests in a couple of my test clouds, or is it only production clouds that you are after
17:26:35 <hogepodge> gema: a capability falls into a category of behavior, like "compute instances", and is a specific test that checks for a capability, like "boot image with hostname"
17:26:49 <hogepodge> gema as many clouds as possible, test or production
17:26:57 <gema> hogepodge: ok
17:27:21 <gema> #action gema to submit a few test runs of refstack and get familiar with it
17:27:33 <hogepodge> We will use the data set (which will be admittedly biased) to give us guidance on what capabilities are commonly in use
17:27:53 <jasonsb> this sounds like another possible application for our introspection thingy
17:27:53 <gema> hogepodge: ok, our test clouds are representative of what we deploy for customers anyway
17:28:02 <jasonsb> so the test results can be supplied along with a description
17:28:11 <hogepodge> gema: if you have a valid tempest configuration refstack should be easy. It only runs tempest.api.* tests
17:28:26 <gema> hogepodge: I have that for icehouse
17:28:34 <gema> hogepodge: I am struggling for kilo, I was going to discuss that later
17:28:42 <hogepodge> jasonsb: No, it just sends an anonymous cloud id.
17:29:02 <jasonsb> hogepodge: ok
17:29:34 <hogepodge> jasonsb: along with a list of all tests that passed (skip and fail results are not reported). It's just a json data structure with no other output from the subunit output or logs
17:30:09 <hogepodge> jasonsb: we will have a user auth system in place soon, though, to help with self-identification
17:30:10 <jasonsb> hogepodge: understood.
17:31:08 <gema> hogepodge: sounds like we should be collaborating and see where we go from there, I cannot talk for the rest of the folks in our group but I will probably start a thread on the mailing
17:31:13 <gema> hogepodge: are you receiving emails?
17:31:58 <jasonsb> gema: agreed.
17:32:07 <hogepodge> gema yes
17:32:23 <gema> hogepodge: good, we'll do that then, see if we can get some traction with the others as well
17:32:36 <gema> hogepodge: thanks for the information
17:33:02 <gema> moving to the next topic then
17:33:07 <jasonsb> hogepodge: one more question
17:33:11 <jasonsb> if its ok
17:33:14 <gema> sure go for it
17:33:23 <hogepodge> that's great, thanks. Your group is an important part of the userbase we want in helping us figure out how to best test clouds. We have a mailing list, defcore-committee, that's had lots of active discussion #link http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
17:33:34 <jasonsb> if you have these test results in place, how do you envision it being used in practice?
17:33:44 <jasonsb> by somebody who owns an application
17:34:02 <hogepodge> jasonsb: we working up a way to pair API calls with tests
17:34:03 <jasonsb> they probably need to understand if their application fits inside defcore i would imagine
17:34:25 <hogepodge> jasonsb: so if an app dev wants to know if their app is interoperable, they should be able to check the test calls to see if they are supported by a cloud.
17:34:45 <hogepodge> whether they are on the defcore list or not
17:35:01 <jasonsb> i see
17:35:14 <jasonsb> that sounds quite interesting
17:35:48 <hogepodge> jasonsb: we're actively pairing tests to api right now #link https://review.openstack.org/#/c/194737/
17:36:25 <hogepodge> one tricky bit is api calls can change depending on cloud config
17:36:41 <hogepodge> nova-net vs the n-permutations of neutron, glance v1/v2, etc
17:36:55 <jasonsb> yes indeed
17:37:11 <jasonsb> and some clients are very slow to update to new api's
17:37:14 <jasonsb> (horizon)
17:37:21 <gema> hogepodge: that's why coverage doesn't really get you that far in terms of interop and system wide testing
17:37:32 <hogepodge> horizon is working on glance v2 right now
17:37:44 <hogepodge> gema: it's a hard problem, indeed. :-(
17:37:58 <gema> hogepodge: it's a fascinating problem
17:38:01 <jasonsb> i wonder if we can work together to fix that
17:38:04 <gema> specially at the scale of openstack :D
17:38:06 <hogepodge> Down the line we may update refstack client to report a test: api dictionary
17:38:13 <jasonsb> audit somethign something
17:39:12 <gema> hogepodge: do you guys have a say in the cyclomatic complexity of calls? i.e. how many levels are too many?
17:39:40 <hogepodge> gema: no, we tend to use whatever the devs have provided
17:39:45 <gema> ok
17:40:14 <jasonsb> interesting stuff.
17:40:19 <jasonsb> gema:  you had next topic?
17:40:25 <gema> yep
17:40:30 <hogepodge> gema: We have a criteria for admitting a test as required: #link https://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst
17:40:43 <hogepodge> thanks everyone for you time
17:40:53 <gema> hogepodge: cool, I will be reading through all of it and asking questions
17:41:11 <gema> hogepodge: thank you for  your time, this has been very educating :D
17:41:28 <gema> #topic Inspector Gadget (automatic service discovery)
17:41:49 <gema> clee: you around? you were going to talk to dkranz about whether redhat were able to contribute
17:41:56 <gema> clee: did you?
17:42:39 <gema> jasonsb: he seems busy, do you have anything on this topic?
17:42:43 <malini> o/
17:42:50 <gema> malini: :D
17:43:01 <malini> better late than never :D
17:43:12 <jasonsb> i dont have anything at moment
17:43:31 <gema> does anyone have any update on the discovery services?
17:43:44 <gema> coolio, that was quick :D
17:43:54 <gema> #topic Open Discussion
17:44:05 <gema> do you guys have any other topic to discuss?
17:44:26 <malini> did we already figure out the chair for next week?
17:44:31 <gema> malini: not there yet
17:44:50 <gema> I was going to bring up the issue of kilo not having a stable version of tempest
17:45:00 <gema> #link http://osdir.com/ml/openstack-dev/2015-05/msg01073.html
17:45:05 <clee> gema: I emailed him but got no reply
17:45:12 <gema> clee: ok
17:45:25 <clee> and he *was* idling in here, but it looks like he's gone now
17:45:37 <gema> clee: ok, I will keep the action there for next week then
17:45:41 * clee nods
17:45:51 <gema> #action  clee ask dkranz if redhat is able to contribute to the inspector gadget concept
17:46:12 <gema> so am I the only one with issues testing kilo with a stable version of tempest?
17:46:13 <malini> we had somebody else from redhat here
17:46:20 <malini> mwagner ?
17:46:41 <malini> gema: probably because nobody else ran tht?
17:46:44 <jasonsb> i am curious about the tooling redhat uses
17:46:53 <jasonsb> and what they might be willing to contribute to tailgate
17:47:00 <gema> malini: indeed
17:47:08 <gema> jasonsb: you can add it to next week's agenda and bring it up
17:47:13 <gema> see if they attend the meeting
17:47:22 <jasonsb> gema: ok
17:47:27 <malini> mwagner_lap: will you have any info on tht?
17:48:04 <gema> malini: he's not present
17:48:10 <gema> hasn't said anything yet :D
17:48:33 <malini> maybe _lap means something
17:48:37 <gema> alright, any other topic?
17:48:51 <malini> gema: I am curious abt the kilo tempest
17:49:00 <gema> malini: what do you want to know?
17:49:08 <malini> is it something we can start to help fix?
17:49:21 <gema> malini: I wanted to talk to matthew and ask him
17:49:34 <gema> malini: because what he said in that email is that there is not supposed to be a stable version of tempest
17:49:53 <malini> hmm…I don't understand that
17:49:59 <gema> malini: but if we are going to use tempest to validate stable versions of clouds, we kind of need a stable version of the test suite as well
17:50:08 <malini> exactly
17:50:10 <gema> malini: they tag old version of tempest on a day
17:50:16 <gema> and they don't really maintain them
17:50:23 <gema> they move on with the development
17:50:28 <gema> that's how I understand it anyway
17:50:33 <gema> and that's why I wanted to talk to him
17:51:35 <gema> we'll have to talk to him and then decide
17:51:43 <malini> thx for the info gema
17:51:46 <gema> maybe if they don't want to maintain stable versions we should do that
17:51:52 <gema> but that is a big commitment
17:52:13 <malini> Openstack supports the past two releases -rt?
17:52:14 <gema> and by maintain I don't mean adding new tests, just keeping the ones there running should be enough
17:52:28 <gema> malini: I think so
17:52:39 <jasonsb> this sounds liek a challenge for defcore
17:52:43 <gema> yet in his email he says: Tempest does not have a stable branch like other openstack projects. For more
17:52:46 <gema> info about that see:
17:52:48 <gema> #link http://specs.openstack.org/openstack/qa-specs/specs/implemented/branchless-tempest.html
17:53:05 <jasonsb> how to give a soft-landing to people who want to test clouds but don't do it often enough to keep up with temptest
17:53:20 <gema> jasonsb: it's not about often
17:53:26 <gema> jasonsb: the tests move on with the development
17:53:33 <jasonsb> true true
17:53:37 <gema> if you don't have a stable version that goes with a particular release
17:53:44 <gema> you are not running the kilo tests
17:53:47 <gema> but something else
17:53:51 <gema> liberty tests against kilo
17:53:53 <gema> sort of
17:54:12 <gema> #action gema to talk to matthew about tempest kilo and report back
17:54:29 <gema> that's all I had
17:54:38 <gema> let's change topic if you guys have nothing else
17:54:50 <gema> #topic Next Week Meeting
17:54:55 <jasonsb> i have one question
17:55:01 <gema> jasonsb: ok
17:55:09 <jasonsb> but its going to sound pretty noob
17:55:15 <gema> you have 1 min
17:55:17 <gema> make it quick
17:55:29 <jasonsb> i was wondering if somebody could give a walkthrough of the rackspace testing
17:55:43 <malini> sure we can
17:55:55 <jasonsb> in general i was thinking we should start to focus on where to start for a core set of tests
17:55:57 <gema> malini: wanna do that next meeting?
17:56:14 <jasonsb> and i thought maybe a walkthrough of rackspace or canonical would be good place to start?
17:56:18 <malini> sure - I also want sam to be here for tht
17:56:35 <gema> #action malini to organise for a walkthrough of the rackspace testing
17:56:41 <malini> I will confirm with sam before committing to the next meeting
17:56:43 <gema> malini: do it for whenever sam can make it
17:56:46 <jasonsb> yay
17:56:50 <gema> we can have that topic on that week
17:56:55 <jasonsb> thnx
17:57:06 <malini> tht is infact a good idea
17:57:15 <gema> now, I won't be here next week
17:57:20 <malini> I guess each of us can walk thru how we test
17:57:21 <gema> any volunteers for chair?
17:57:30 <gema> malini: yep, sounds good
17:57:42 <malini> I am tentative next week - but will try to be here
17:57:52 <gema> jasonsb: will you be here for sure?
17:58:00 <murphyslawbbs> sorry im so late :). I will resend invite for next week
17:58:10 <jasonsb> yes
17:58:16 <gema> jasonsb: wanna chair?
17:58:18 <jasonsb> week after i'm out though
17:58:19 <jasonsb> ok
17:58:26 <malini> I can do week after tht
17:58:41 <gema> #action jasonb to chair on July 9th
17:58:50 <gema> #action malini to chair on July 16
17:59:31 <gema> sounds like we are done
17:59:36 <gema> #endmeeting