18:02:04 <vipul> #startmeeting Cue
18:02:04 <openstack> Meeting started Tue Apr  7 18:02:04 2015 UTC and is due to finish in 60 minutes.  The chair is vipul. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:08 <openstack> The meeting name has been set to 'cue'
18:02:14 <vipul> yay our first meeting1
18:02:32 <davideagnello> hello!
18:02:42 <vipul> let's wait a few minute for people to trickle in
18:02:43 <esmute> Hello Cueans
18:03:03 <esmute> How do you call people in the Cue community?
18:03:25 <vipul> that's a great topic
18:03:37 <ekarlso> cuebans
18:03:39 <ekarlso> :P
18:03:56 <vipul> #link https://wiki.openstack.org/wiki/Meetings/Cue
18:04:27 <esmute> ekarlso: I'd to have what you are smoking
18:04:57 <vipul> #topic kick-off
18:05:21 <vipul> Ok so since this is our first meeting, i'll go over how these are typically run
18:05:34 <vipul> obviously we'll be in this channel every Tuesday at this time
18:05:47 <sputnik13> first meeting w00t
18:05:53 <vipul> over the week, feel free to add to the agenda at https://wiki.openstack.org/wiki/Meetings/Cue
18:06:00 <vipul> heya sputnik13
18:06:24 <sputnik13> hi :)
18:06:46 <vipul> remember agenda items can be anything
18:06:56 <vipul> even things like links to reviews you want folks to look at
18:07:16 <sputnik13> ok
18:07:37 <vipul> We'll talk about each item on the agenda, and if there are any actions that come out of it, we'll record them
18:07:41 <sputnik13> is dan here?
18:07:57 <vipul> nope sputnik13
18:08:07 <sputnik13> I'll go poke him, brb
18:08:30 <vipul> At the following week's meeting, we'll review progress on the things that were actioned
18:08:52 <vipul> makes sense?
18:08:59 <davideagnello> yup
18:09:03 <sputnik13> sounds good
18:09:14 <vipul> abitha: you're quiet ;0
18:09:17 <vipul> ;)
18:10:11 <abitha> ya :)
18:10:25 <vipul> #topic Integration Testing
18:11:21 <sputnik13> vipul is busy..  davideagnello you want to start on integration testing?
18:12:04 <esmute> Hi danielallegood
18:12:13 <danielallegood> Hi
18:12:13 <sputnik13> hey danielallegood
18:12:14 <sputnik13> :)
18:12:24 <davideagnello> yes, so I have been assessing two frameworks for use in intergration and scenario testing in Cue
18:12:28 <vipul> sorry.. got pulled away in a side conversation
18:12:30 <davideagnello> these are Tempest and Rally
18:13:24 <esmute> #link https://wiki.openstack.org/wiki/Rally
18:13:29 <davideagnello> both frameworks are intended for OpenStack integration/scenario tests and are being used by various openstack services.
18:13:44 <davideagnello> some services like Neutron, make use of both
18:14:23 <vipul> any notable differences that you've noticed.. just off the cuff?
18:14:46 <davideagnello> the intent of these frameworks is to test a service from the public API (customer facing interfaces) to ensure the behavior is as expected, stress tests, benchmarking, etc..
18:14:48 <davideagnello> yes
18:15:08 <davideagnello> I am still evaluating Rally, but so far the differences I have noticed are
18:15:43 <davideagnello> rally provides automation to deploy your testing environment as part of the provisioning of tests on a openstack environment
18:16:51 <davideagnello> Rally makes it easier for benchmarking and writing tests that load your service by running the same test a specified number of times, captures run time results,
18:17:32 <davideagnello> the framework allows for a more modular test architecture, thereby increasing code reuse
18:17:46 <davideagnello> this is evident with the concept of plugins
18:18:45 <vipul> We talked about this in another meeting, and you mentioned the principle for both was to run tests are black-box
18:18:51 <esmute> How easily (or complex) is to integrate rally with the openstack-ci?
18:19:08 <vipul> how / where would we run non-black-box tests
18:19:13 <vipul> lots of hyphens
18:19:29 <davideagnello> yes, the principle of both frameworks are not to reach into the service but interface with it as the customer would
18:19:30 <sputnik13> I think blackbox can be one word :)
18:19:41 <vipul> sputnik13: :P
18:20:09 <davideagnello> Tempest requires your tests hit the REST API directly, without any client code
18:21:16 <sputnik13> davideagnello: do we have example test cases for tempest and rally to see which is easier to work with?
18:21:23 <davideagnello> both framework repos include test code for the various openstack services
18:22:21 <vipul> esmute: A bunch of projects are already integrating rally-jobs as part of openstack-ci, so i think it shoudl be as simple as tempest
18:22:29 <davideagnello> no for cue, I have been looking at existing tests for both rally and tempest.  Rally seems easier to work with but it will become more evident when I write some example tests with the framework of choice
18:23:13 <vipul> Ok thanks davideagnello looks like we still need to do some more prototyping before we go with one or the other
18:23:47 <vipul> #action davideagnello to discuss cue's integration testing approach
18:24:04 <sputnik13> davideagnello: it might prove very instructive/revealing to try and write your own, some sort of simple test in both frameworks, to see how easy it is to find documentation for the features you need, etc
18:24:05 <davideagnello> yup, I am prototyping these as my next task
18:24:23 <vipul> +1
18:24:57 <vipul> sputnik13: How is progress on the end-to-end taskflow / conductor tests
18:25:16 <davideagnello> yes, I was going to do it for the one we chose to based on evaluating existing tests in both but I can write sample tests for both
18:26:12 <sputnik13> vipul: progress is slow, I'm thinking that a conductor test is needed and an external test (preferably with something like rally/tempest) needs to drive the end-to-end
18:26:45 <sputnik13> we have tests that bypass the conductor to pull jobs from the jobboard and execute, but no tests to run the conductor
18:27:10 <vipul> are there issues running these as functional tests ? assuming you get zeik (however that is spelled) integrated?
18:27:13 <davideagnello> sputnik13: we currently have a test or two which posts a task to the in memory zookepper (zule), can't a worker pull the job the execute it?
18:27:29 <sputnik13> vipul: zake :)
18:28:25 <sputnik13> we could do an end to end functional test but that looks more like an integration test to me given worker and api are normally separate processes
18:29:16 <vipul> sputnik13: yea that's a good point..
18:29:40 <vipul> is there a way to invoke the conductor directly in a test?
18:29:58 <vipul> or is that already happening
18:31:36 <sputnik13> well, thinking about it, the conductor needs to be started and run as a separate thread/process, its main execution code is a persistent infinite loop, not much other code to be exercised through unit/functional testing
18:32:26 <sputnik13> hmm, persistent infinite...  >_<
18:32:42 <esmute> just to be clear, what is the main purpose to test the conductor? Is this something that we code in cue?
18:33:00 <esmute> isnt posting a job and retrieving it enough?
18:33:11 <vipul> #link https://github.com/stackforge/cue/blob/master/cue/taskflow/service.py
18:33:16 <vipul> esmute: ^
18:33:18 <sputnik13> esmute: the conductor itself isn't our code, it's provided by taskflow
18:33:38 <vipul> sputnik13, esmute : I suppose what we want to cover is the wrapper
18:33:42 <sputnik13> so, the proper test is perhaps to ensure we're spinning up conductor properly
18:34:19 <davideagnello> that makes sense
18:34:33 <sputnik13> and so spin up conductor using our taskflow service code, submit a simple job, and make sure it's consumed
18:34:42 <vipul> sputnik13: Could we create a fixture that basically invokes start()?
18:35:05 <sputnik13> vipul: sure, although I think we could just as well create a service instance and start it
18:35:51 <sputnik13> fixtures would be external dependencies like zookeeper or nova-api
18:36:04 <davideagnello> is this a real zookeeper vs zule?
18:36:11 <vipul> zake davideagnello  ;)
18:36:16 <sputnik13> davideagnello: s/zule/zake
18:36:26 <sputnik13> zuul is a different beast :)
18:36:33 <esmute> +1. yeah. im assuming that the taskflow team already have tests for the conductor... So just testing that the conductor config, that starts and that it is able to handle job requests, should be sufficient
18:36:47 <davideagnello> sputnik13: vipul:  zake!
18:36:49 <sputnik13> esmute agreed
18:37:12 <sputnik13> also, I realized, we have no unit tests for the functions provided in service.py
18:37:43 <sputnik13> sorry, client.py
18:37:48 <sputnik13> like get_client_instance
18:37:56 <sputnik13> or create_persistence
18:38:01 <sputnik13> #link https://github.com/stackforge/cue/blob/master/cue/taskflow/client.py
18:38:02 <vipul> Yea, that's another can of worms that needs to be tested
18:38:07 <vipul> it does a lot!
18:38:23 <sputnik13> so I guess my action just expanded in scope :)
18:38:49 <davideagnello> @sputnik13 so we have a test that posts a job to the jobboard and verifies the count in jobboard after the post, is this the same?  or do we want a worker/conductor to take the jobs and do something with it?
18:39:35 <davideagnello> sputnik13: we are testing client.post
18:39:58 <sputnik13> davideagnello: there should be a separate unit level test for client.post, yes
18:40:08 <davideagnello> sputnik13: ok
18:40:43 <sputnik13> and there should be a service.py test that ensures we spun up conductor properly by posting to jobboard and ensure that it was consumed
18:40:55 <sputnik13> actually, I think this is a perfect place for the listener functionality that taskflow provides
18:41:20 <vipul> can you elaborate?
18:41:21 <sputnik13> since conductor is running as a separate process/thread we can't tell when the consume happens respective to when we post
18:41:42 <sputnik13> so in tests normally you're taking some action and verifying the consequence
18:42:11 <sputnik13> since the conductor is a separate thread/process from the one running our test we can't tell when the consume happens
18:42:26 <sputnik13> which would be the consequence we want to verify
18:43:02 <vipul> I suppose you could check tthe thing you have a handle on, in this case, the client connection to zake and ensure the state of the job has changed
18:43:26 <sputnik13> listener/notifier is functionality provided by taskflow to have callbacks that run in between state changes
18:43:32 <vipul> but yes, the flow is running as a separate thing.. so you couldn't do much beyond that
18:44:01 <sputnik13> right, I think we could leverage the listener/notifier feature, or at minimum do a busy loop for some number of seconds to make sure the consumption happens
18:44:23 <vipul> ok cool.. good stuff
18:44:50 <vipul> assuming we're done with this topic.. moving on?
18:44:56 <esmute> Should we create BP to discuss this further?
18:45:11 <sputnik13> uhh, bp seems over the top :)
18:45:33 <esmute> it seems there are a lot of things that we can do and tests... so it is hard (at least for me) to know what is a good testing solution vs another one
18:45:35 <vipul> not sure this is deserving of a BP.. using the listener for other things beyond tests yes
18:45:49 <sputnik13> I think BPs are only for feature development, no?
18:46:02 <esmute> ok.. maybe a wiki then.. Something that is not irc chat logs :)
18:46:05 <sputnik13> I agree capturing test approaches is a good thing
18:46:08 <vipul> typically yes
18:46:11 <sputnik13> yes wiki would be good
18:46:26 <sputnik13> or even developer documentation would be good too
18:46:27 <sputnik13> under docs
18:46:33 <vipul> #action sputnik13 to document the approach for testing end-to-end conductor as a functional test
18:46:49 <vipul> ok moving on..
18:46:55 <vipul> #topic open discussion
18:47:03 <esmute> or we can also use the feature specs that other services use
18:47:37 <vipul> esmute: so what folks tha work on cue called.. besides cuebans
18:47:49 <esmute> Have this in rst files so that we can review/add comments via gerrit
18:48:05 <vipul> umm.. that's a lot of process to take on..
18:48:16 <vipul> i dunno how i feel about that yet..
18:48:27 <esmute> ok.. we can starts with wiki then.. and add comments in the wiki
18:48:51 <vipul> yea let's keep it simple.. we wnat to document this stuff publicly.. but move fast
18:49:36 <vipul> anyone have stuff to bring up?
18:49:50 <vipul> who's going to the summit?
18:50:21 <davideagnello> -1 visa issues :(
18:50:36 <esmute> i'll be there... if nothing changes
18:50:42 <sputnik13> I'll be there
18:50:48 <esmute> ill be there the weekend before anyways.. for U2 concert
18:50:59 <vipul> yep, me too.. (not the concert)
18:51:18 <vipul> hopefully we meet some folks that are interested in this project there
18:51:33 <esmute> vipul, do we have a design meeting at the summit?
18:51:43 <esmute> or something similar.
18:51:52 <vipul> nope.. no official slot.. we'll have to figure something out so people can find us
18:52:12 <sputnik13> vipul we should also be going to other projects, I'm probably going to stop in at the oslo meetings
18:52:19 <sputnik13> help spread the word :)
18:52:21 <esmute> i wonder if we can crash at the marconi's design summit
18:52:23 <esmute> :p
18:52:30 <vipul> sputnik13: +1
18:52:56 <vipul> esmute: I'll talk to their PTL to see if they have any slots they don't plan to use
18:53:21 <vipul> Oh..
18:53:29 <vipul> sputnik13: what's the status of our stickers
18:53:42 <sputnik13> zaqar is another message queue...  should cue provision zaqar?
18:53:43 <sputnik13> :-D
18:54:04 <sputnik13> vipul: need to check in with my friend about that, he went silent, I think he might be sick
18:54:08 <davideagnello> sputnik13: hahaha
18:54:13 <sputnik13> last time I talked with him he was working from home
18:54:26 <vipul> ok.. we gotta sport those at the summit ;)
18:54:33 <sputnik13> I'll take an action to follow up
18:54:52 <vipul> #action sputnik13 to give status on stickers for Cue
18:55:06 <vipul> Good first meeting guys.. we used the whole hour :D
18:55:43 <vipul> sputnik13: remember we're not a project unless we have t-shirts or stickers
18:55:44 <sputnik13> yay!  my meeting quota is completely full now :)
18:55:54 <sputnik13> oh, you didn't mention t-shirts...
18:55:57 <sputnik13> should we do t-shirts?
18:56:02 <sputnik13> at least for the team?
18:56:02 <davideagnello> yes!
18:56:12 <vipul> i think a few maybe
18:56:21 <esmute> yeah.. we can wear during the summit... it will help to spread the word around
18:56:24 <vipul> not sure we can do enough for the summit
18:56:57 <sputnik13> well, just enough for the team members then, it should help with visibility like esmute says
18:56:58 <esmute> It gotta be cool and something that we can wear outside...
18:57:06 <esmute> No reference to starwars or something
18:57:14 <vipul> +1
18:57:17 <sputnik13> what, what's your beef with starwars
18:57:22 <vipul> sputnik13: run t-shirt design choices by esmute  ;)
18:57:35 <sputnik13> I think you just alienated half the world of geekdom
18:57:36 <sputnik13> :)
18:57:45 <esmute> sputnik13: Nothing against starwars.. i just want a tshirt that i can wear not just when i sleep
18:57:51 <sputnik13> hahah
18:57:52 <vipul> LOL
18:58:11 <sputnik13> yes, a t-shirt my wife won't automatically ban would be good
18:58:12 <sputnik13> :)
18:58:20 <sputnik13> agreed
18:58:29 <vipul> coolio.. see ya'll next week
18:58:32 <vipul> #endmeeting