22:00:59 <mestery> #startmeeting networking_third_party_testing
22:01:00 <openstack> Meeting started Wed Jan 15 22:00:59 2014 UTC and is due to finish in 60 minutes.  The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:02 <lyxus> Hello, yes I am
22:01:04 <openstack> The meeting name has been set to 'networking_third_party_testing'
22:01:19 <mestery> #link https://etherpad.openstack.org/p/multi-node-neutron-tempest Neutron 3rd Party and Muilti-Node Testing Etherpad
22:01:35 <mestery> We'll use the etherpad for notes today and to update progress, so please sign in with your name there if you can.
22:02:00 <krtaylor> o/
22:02:36 <mestery> So, first off, has anyone succesfully completed the entire 3rd party testing requirement? :)
22:03:19 <luqas> not yet :-(
22:03:35 <luqas> still working on it
22:03:38 <mestery> luqas: I suspect most are still in various stages here, you're not alone.
22:03:51 <hemanthravi> partially, not sure if we are on the right track
22:04:06 <hemanthravi> adding what we did to the etherpad
22:04:20 <hemanthravi> for comments
22:04:22 <mestery> hemanthravi: Thanks!
22:04:33 <mestery> So, banix, you had some questions you posed to the list, should we start with those for discussion now?
22:04:52 <banix> Is the instructions mentioned towards the end of http://ci.openstack.org/third_party.html a good place to start?
22:04:57 <banix> mestery: sure
22:05:39 <banix> I have followed the instructions there: So have a Jenkins running with Gerrit plugin
22:06:01 <mestery> banix: Yes, I believe those are the instructions folks have been and should be following as a starting point.
22:06:33 <banix> The instructions for configuring Jenkins are reasonable. The part I am not sure about is the reporting back to Gerrit
22:06:59 <mestery> banix: OK, what's got you hung up there?
22:07:22 <banix> last line of the instructions on the web page above says: "This job will now automatically trigger when a new patchset is uploaded and will report the results to Gerrit automatically."
22:08:09 <mestery> So, I've seen a few other testing rigs reporting back, can anyone comment on banix's question here that has that part working already?
22:08:14 <banix> So I am assuming if all the steps in the Build phase return without error a +1 is reported?
22:09:09 <hemanthravi> banix: do the jenkins steps I added to the etherpad seem right?
22:09:38 <banix> hemanthravi: let me have a look please
22:10:43 <banix> I believe you can do the step 5 items in different way; what hemanthravi has outlines is one way of doing it
22:11:27 <banix> Alternatively, you could have a multi-node setup, and all that but I believe for the first try those steps sound reasonable
22:11:55 <mestery> I agree, steps 5 could differ slightly, but the general approach looks reasonable to me as well.
22:11:57 <hemanthravi> for 5.5, do we specify the list of tests to be run or is there a standard set of test
22:12:07 <banix> I am not sure about how to do step 6. Being cautious to avoid reporting/voting incorrectly
22:12:35 <mestery> hemanthravi: I think you can run the standard tempest tests.
22:13:02 <banix> hemanthravi: I would think the tests could be limited to say your plugin only
22:13:46 <hemanthravi> remember some disc about filters for the tests to be run, is this a config outside jenkins?
22:14:22 <banix> lugas: are you setup the system that posts reviews for Midokura?
22:14:36 <shivh> Currently there is push to consolidate all network tests in one directory, one could specify that to nosetests to run networking tests (suggestions?)
22:14:37 <banix> hemanthravi: no you can specify the directories
22:14:52 <luqas> yes, but currently I fake the answer to post +1 votes always
22:14:52 <mestery> You can decide which subset of Tempest tests to run to work your plugin out
22:15:06 <hemanthravi> got it, thanks
22:15:23 <luqas> you can configure that in the gerrit trigger plugin
22:15:26 <banix> luqas: how do you do that? where do you specify that?
22:15:49 <banix> in the gerrit plugin and not the Jenkins job configuration?
22:16:05 <shashank_> So we've noticed that devstack is a bit flaky. So I was wondering is it ok to vote -1 if devstack fails?
22:16:40 <mestery> shashank_: What do you mean devstack is flakey? devstack itself fails?
22:16:47 <shashank_> yep
22:17:10 <mestery> shashank_: That shouldn't happen, I think you need to figure out why devstack is failing, maybe it's in your local.conf file or something.
22:17:42 <shashank_> We were actually hitting a db migration bug
22:17:52 <shashank_> We worked around it by enabling lbaas
22:18:15 <mestery> shashank_: Cool! So maybe that was your problem, and not a devstack issue itself?
22:18:36 <banix> shashank_:  yes that is not a devstack issue
22:18:42 <shivh> mestery: devstack to use should be stable version, not master?
22:18:56 <banix> shashank_: I think the fix is submitted but not merged or may be got merged recently
22:19:06 <mestery> shivh: I would suggest the latest, e.g. track upstream. But maybe I'm missing a nuance somewhere. :)
22:19:50 <shashank_> My concern was what if such bugs creep into devstack and cause stack to fail
22:19:52 <luqas> banix: in manage jenkins -> gerrit trigger
22:20:04 <shivh> mestery: top of the master (head) may be unstable, somewhere there should be a good known version (stable)
22:20:06 <luqas> you set the reporting values
22:20:33 <mestery> shivh: True, but you also run the risk of an older version of devstack not being able to support deploying the latest code, right?
22:20:44 <shashank_> So I was a bit concerned about stacking before testing each patch
22:21:07 <shivh> mestery: I see your point. maybe head is fine to use.
22:21:08 <banix> luqas: I see that; so you set the value for successful, etc; but how is it decided that the test was successful is I guess what I am missing
22:21:22 <roeyc> shashank_: You can vote 0 and just report with a message.
22:21:37 <mestery> shivh: I think there's a balance, and if you look at what -qa and -infra do, they have to strike that balance all the time.
22:21:39 <shashank_> cool…
22:21:47 <mestery> shivh: But I think this is something we need to all be aware of.
22:22:00 <shashank_> @banix, it depends on the exit value of the script that you run
22:22:12 * mestery nods in agreement with shashank_.
22:22:37 <banix> I see. Thanks shashank and luqas
22:23:22 <banix> So I think just to be safe, one could set the values for reporting to 0 or +1 for all cases
22:23:42 <banix> Just don't want to incorrectly vote -1 on patches and block their review
22:24:28 <mestery> banix: Yes, we've seen that happen already. But I think -infra is limiting the voting ability of the testing setups until they can prove themselves, which will alleviate that problem as well.
22:25:02 <banix> mestery: That's good to know!
22:25:21 <mestery> banix: :)
22:25:52 <luqas> regarding the tempest test, network related is enough?
22:26:54 <banix> With respect to testing a plugin, I would thing any change to neutron directory could lead to failure and therefore should trigger a test but I am thinking that to begin with I will limit the trigger to the plugin I am concerned with.
22:27:18 <shashank_> Yeah… Even I am curious as to which tests need to be run
22:27:38 <mestery> So, those are differences: Waht tests to run, and what changes trigger the running of the tests.
22:27:40 <shivh> so voting with 0 is allowed?
22:27:52 <mestery> The latter was suggested as changes to your plugin/MechanismDriver along with all Jenkins changes.
22:27:58 <mestery> The former woudl be the Tempest tests, or a subset of those.
22:28:03 <mestery> Makes sense?
22:28:04 <banix> That would be another way of limiting the impact of my voting….
22:28:37 <banix> mestery: where/what are Jenkins changes?
22:28:47 <mestery> I don't think the testing rigs voting "0" would have much value, unless it was only temporary.
22:29:01 <mestery> banix: I mean, changes submitted by the Jenkins user (e.g. translation changes as an example).
22:29:23 <banix> mestery: thanks;
22:29:49 <mestery> banix: Cool.
22:30:14 <banix> so that means we can trigger by who submitted the patch set?
22:30:34 <mestery> banix: I believe so, yes.
22:33:07 <mestery> OK, so is there anything else people want to discuss here?
22:33:58 <hemanthravi> is there a criteria to decide, when the test setup can start voting?
22:34:45 <mestery> hemanthravi: That's a very good question actually. I think the idea is that the test rig has to be running ok for a while, a contact from that company/project has to be particiapting upstream in meetings.
22:34:58 <mestery> Thsoe are the two things which I remember from an email to the list a while back.
22:36:56 <banix> So, multi-node testing would come later on? Anybody working on that?
22:37:49 <mestery> banix: That's ... complicated.
22:37:55 <banix> :)
22:38:00 <mestery> As anita indicated in email, that doesn't exist yet upstream.
22:38:07 <mestery> All gate testing is done on a single node if you can believe it!
22:38:22 <mestery> Now, I see no reason why this shouldn't work, after all, Tempest runs against an OpenStack cloud.
22:38:38 <mestery> There may be issues with deploying this for testing, etc. But that's all things we can work through together perhaps.
22:38:40 <banix> I see
22:39:15 <banix> sounds good
22:40:04 <mestery> OK, good. :)
22:40:17 <mestery> banix: There could be problems, but if there are, lets all work through them together.
22:40:33 <banix> mestery: Yes of course.
22:40:59 <banix> So I think from what I have heard so far, Jenkins with Gerrit plugin seems the way to go.
22:41:06 * mestery nods.
22:41:08 <mestery> I think so.
22:41:30 <luqas> a last question regarding the stability of devstack
22:42:04 <luqas> sometimes when stacking, at the end, it gives some errors, when setting up the default config
22:42:07 <shivh> mestery: thanks for the clarifications - much needed.
22:42:27 <luqas> can it be because of some race condition?
22:43:41 <mestery> luqas: I would verify it's not related to your plugin, try ensuring you can get consistent devstack runs outside of the test harness.
22:43:46 <mestery> OR are you saying you've already done that?
22:44:39 <shivh> if there are race conditions others may have experienced it. Maybe ask the question on ML.
22:44:46 <luqas> it happens sometimes with our pluggin that I have to unstack a couple of times before everything is setup correctly
22:45:10 <mestery> Good point shivh.
22:45:12 <luqas> I have to be sure that this has never happen of plain devstack without our plugin
22:45:23 <mestery> luqas: That would be a good datapoint to have.
22:45:42 <mestery> FYI: I am consisently stacking with the latest master and Neutron and I've never seen it fail with a race condition.
22:45:45 <luqas> mestery: ok, i?ll check that
22:45:51 <mestery> luqas: Cool.
22:47:24 <mestery> OK, anything else before we wrap this meeting up?
22:48:35 <mestery> OK, lets keep communicating in channel (#openstack-neutron and #openstack-qa) and on the ML.
22:48:50 <mestery> Thanks everyone, and good luck as we move to the 3rd party testing deadline next week!
22:48:52 <mestery> #endmeeting