18:01:06 <jaypipes> #startmeeting third-party-ci
18:01:07 <openstack> Meeting started Mon Mar  3 18:01:06 2014 UTC and is due to finish in 60 minutes.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:01:10 <openstack> The meeting name has been set to 'third_party_ci'
18:01:32 <jaypipes> Welcome all.
18:01:37 <jaypipes> I will be taking some notes on this etherpad:
18:01:38 <jaypipes> #link https://etherpad.openstack.org/p/third-party-ci-workshop
18:01:54 <jaypipes> as we answer questions, we will put the question and answer on that etherpad.
18:02:17 <jaypipes> I'd like to ask up front that we focus on answering *one* question at a time, please.
18:02:32 <jeblair> for folks who haven't used etherpad before, if you clirk the "person" icon in the top right, you can enter your name, then when you make edits, we can see who edited what
18:02:44 <jeblair> 'click', not 'clirk'
18:02:51 <jaypipes> I'm going to go in order, with trinaths's questions, then roeyc, then VinayMahuli .
18:03:02 * jaypipes clirks on jeblair
18:03:11 * jeblair makes a clirking sound
18:03:39 <jaypipes> OK. Here's the first question, from Trinath's ML post:
18:03:41 <jaypipes> [1] sandbox-dvsm-tempest-full runs all the test cases where few fail, causing the completed build to fail.
18:03:41 <jaypipes> How to restrict the testcases?
18:04:01 <trinaths> okay
18:04:48 <jaypipes> trinaths: OK, first things first, let's please refer to this as just "tempest-dsvm-full" (as it has been renamed in the upstream repository to better reflect that the job is not restricted to the sandbox project)
18:04:48 <ArxCruz> You can edit the tox.ini from /opt/stack/new/tempest/ to add tests you want to skip
18:05:29 <jaypipes> ArxCruz:one sec...
18:05:36 <ArxCruz> jaypipes: :-#
18:05:47 <jaypipes> #link https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/examples.yaml#L11
18:06:02 <jaypipes> s/tempest-dsvm-full/ddsvm-tempest-full/ ... sorry
18:06:32 <trinaths> jaypipes: okay.. let me search the same
18:06:38 <jaypipes> trinaths: so, Tempest can run any number of subsets of tests
18:07:33 <jaypipes> trinaths: what you need to find out is how to indicate to tempest which of these tests get run, and do this using Bash scripts in your Jenkns jobs configuration. So, let's show how to do that.
18:07:37 * jaypipes grabs a link...
18:08:02 <trinaths> jaypipes: one Question here: since tempest-full is meant to run any number or test cases, then how do we restrict to few.. like skip cinder and swift like so..
18:08:12 <trinaths> jaypipes: okay please go ahead..
18:08:25 <jaypipes> trinaths, all: OK, please go here: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L326
18:08:27 <jaypipes> #link https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L326
18:09:09 <jaypipes> This is the script that runs the devstcak-based tests in the gate, and from that line and below, you will see different *invocations* of running tempest *using the tox CLI tool*
18:09:21 <jeblair> sdague: ^ do we need a devstack-gate parameter to make it easier to configure what tests tempest runs?
18:09:39 <jeblair> sdague: i think some stackforge projects could use that too....
18:10:00 <jeblair> (this might be off-topic, we can take it into #openstack-infra)
18:10:05 <mtreinish> jeblair: it'll probably be helpful
18:10:08 <jaypipes> jeblair: that would make things easier/more flexible, but looking at the devstack-gate script is a good place to see how everything works (for folks running their own scripts..)
18:10:23 <mtreinish> the tox all job can take a regex but we don't have a way to do that with devstack-gate
18:10:37 <jeblair> jaypipes: yeah, i was thinking that 3rd party folks might be able to reuse devstack-gate more directly if we did that
18:10:49 <jaypipes> trinaths: so, as an example, having the DEVSTACK_GATE_TEMPEST_TESTR_FULL env variable set to 1, would execute this little guy:
18:10:49 <jaypipes> sudo -H -u tempest tox -etestr-full -- --concurrency=$TEMPEST_CONCURRENCY
18:11:04 <ArxCruz> mtreinish: jeblair jaypipes what we are doing in our side, is switch the tox.ini adding the jobs we want to skip
18:11:52 <jaypipes> ArxCruz: that is also a perfectly fine method of configuring tempest. I presume you are doing some sort of sed/awk magic to add/replace things in the tox.ini file?
18:12:09 <trinaths> jaypipes: true.. ! i understand.. we need to configure them as 'export' vars .. for automated jenkins jobs
18:12:24 <ArxCruz> jaypipes: we configure our jobs to copy our tox.ini from our git repo before it runs
18:12:34 <ArxCruz> for example, in [testenv:full] section of tox.ini we have this:
18:12:42 <ArxCruz> bash tools/pretty_tox.sh '(?!.*(test_volume_backup_create_get_restore_delete|\[.*\bslow\b.*\]).*)(^tempest\.(api|scenario|thirdparty|cli)) {posargs}'
18:12:53 <ArxCruz> this will skip the test_volume_backup_create_get_restore_delete test
18:13:02 <jaypipes> trinaths: right. all of the DEVSTACK_GATE_TEMPEST_XXX variables can be set in your Jenkins job configuration for all of those individual blocks starting at line 326.
18:13:19 <anteaya> ArxCruz: can we teach those attending good irc habits by using paste.openstack.org? they may not know how
18:13:22 <jaypipes> trinaths: and jeblair and sdague are discussing adding more of those types of variables for third-party testing uses.
18:13:37 <ArxCruz> anteaya: oh, sure
18:13:41 <anteaya> thx
18:13:49 * jaypipes making a couple notes in etherpad...
18:14:10 <ArxCruz> jaypipes: here's our tox.ini http://paste.openstack.org/show/71835/
18:14:21 <anteaya> ArxCruz: :D
18:14:28 <ArxCruz> we are skipping some jobs that's unstable in our environment
18:14:46 <mtreinish> ArxCruz: I would avoid doing that in the tempest tox.ini because that file changes somewhat regularly. So you may hit an issue because of that eventually.
18:15:43 <trinaths> jaypipes: also, the script its self writes an localrc file with these parameters. Can we modify .localrc.auto file for these required changes .. Any comments ??
18:15:46 <ArxCruz> mtreinish: what you suggest then ?
18:16:19 <mtreinish> ArxCruz: the tox 'all job' can take a regex as a posarg and work as expected
18:16:44 <mtreinish> so you can run tox -eall -- 'regex filter' and it should work the same
18:17:20 <jaypipes> mtreinish: ++ would you mind adding that note to the etherpad?
18:17:41 <trinaths> jaypipes: in the last statement you mean to say, that from the browser while configuring the jenkins job we need to inject the env variables.. ??
18:17:52 <mtreinish> jaypipes: sure, I'm also drafting a devstack-gate patch per your suggestion to add that functionality to d-g
18:18:12 <jaypipes> mtreinish: awesome, ty
18:18:32 <jaypipes> trinaths: yes. note here: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/examples.yaml#L33
18:18:56 * jeblair stops drafting a devstack-gate patch
18:19:00 <jaypipes> trinaths: that we export an environ variable. That would be where you would set the appropriate DEVSTACK_GATE_TEMPEST_XXX variable.
18:19:12 <jaypipes> trinaths: acutally, can be more than one of those variables...
18:19:26 <jaypipes> trinaths: for instance, if you only wanted to run the Tempest smoke tests, you would put:
18:19:53 <jaypipes> export DEVSTACK_GATE_SMOKE_SERIAL=1
18:20:02 <jaypipes> trinaths: ^^ for the serial-running smoke tests...
18:20:19 <trinaths> jaypipes: okay... got it..
18:20:30 <trinaths> jaypipes: its clear
18:21:18 <jaypipes> trinaths: and what mtreinish, jeblair and others having been talking about is adding more flexibility to that part of the devstcak-gate-vm.sh script to allow things like custom regexes, etc.
18:21:59 <jaypipes> trinaths: OK to move on to your second question?
18:22:03 <ArxCruz> jaypipes: mtreinish so, the -efull also have this functionality ? or just -eall ?
18:22:03 <trinaths> jaypipes: okay.. I'm viewing the etherpad.. for the same..  that really helps me ..
18:22:19 <VinayMahuli> To access gerrit from another host in a network we need to put gerrit.war web app in a web server called Jetty or tomcat.       I tried to integrate gerrit.war with tomcat, but it didn't work for me (on tomcat, the web app gerrit.war got deployed, but wasn't able to access it from the browser). Can you give some pointers on deploying gerrit.war in a tomcat webserver?
18:22:38 <trinaths> "tox -eall -- 'regex filter'" where to specify this command.
18:22:50 <jaypipes> ArxCruz: -efull and -eALL are different, IIRC. -eALL is just telling tox to execute the tests that are specified in the tox.ini section to represent "ALL tests"
18:23:07 <anteaya> I have just made a new heading on the etherpad for Questions
18:23:22 <jaypipes> trinaths: I believe we will be adding a way to specify the regex in an environ variable.
18:23:28 <anteaya> those with questions please post them there and jaypipes will select them one at a time
18:23:40 <anteaya> VinayMahuli: can you do that?
18:23:46 <VinayMahuli> sure
18:23:51 <jaypipes> trinaths: so that all you would do is export DEVSTACK_GATE_TEMPEST_REGEX or something like that, in your Jenksin Job config.
18:24:08 <akerr> jaypipes: question —  this may be specific to cinder certification, or maybe apply to other projects as well:  Is the idea of 3rd party testing to only test the "common" community tempest tests and report that those work; or should we run against the community tempest tests as well as our own internally customized tests (that exercise vendor specific functionality) and report +/- on those combined results?
18:24:10 <jaypipes> trinaths: OK, I am moving on to your next question now.
18:24:15 <ArxCruz> jaypipes: ok
18:24:20 <trinaths> jaypipes: sure .. please go ahead
18:24:22 <jaypipes> akerr: noted. will answer in order.
18:24:44 <jaypipes> [2] After the build is failed, still services like nova, swift are running. Does Jenkins using devstack unstack the stack.
18:24:50 <jaypipes> OK, this one is easy :)
18:25:37 <jaypipes> trinaths: so, as you correctly note, these services remain running (as devstack controls the screen sessions in which those services are running)
18:26:31 <trinaths> jaypipes: Here, I want understand that, so these services affect the next patch set under test ?
18:26:39 <jaypipes> trinaths: Jenkins does not stop those services, nor run unstack.sh, AFAIK. Though there may be upgrade (grenade) scenarios where that might happen. Could jeblair, sdague  or dtroyer comment on that?
18:27:24 <sdague> devstack-gate doesn't unstack
18:27:24 <clarkb> jaypipes: trinaths: we run all of our tests on single use slaves that are deleted after the tests complete. As a result the tests don't stop services
18:27:26 <sdague> we assume you create a clean environment before it runs
18:27:28 <sdague> and that the state at the end is throw away
18:27:41 <trinaths> when a old build fails, all the openstack services run as they are.. when a new patch arrives for test, devstack will start from 'git clone' ... till the end.. will this not affect the test.
18:27:59 <clarkb> trinaths: it won't because we never run tests on a machine more than once
18:28:22 <jaypipes> clarkb: trinaths is asking for non-single-use slaves :)
18:28:31 <trinaths> okay.. but how do we clean the environment used for previous test
18:28:33 <clarkb> jaypipes: there is no such thing with upstream devstack testing
18:28:42 <jaypipes> clarkb: understood.
18:28:44 <trinaths> jaypipes:  true ..
18:29:03 <luqas> anteaya: can you share the etherpad link please, I was late
18:29:10 <jaypipes> clarkb: however, what trinaths is asking is for non-single-use slaves, whether an unstack.sh would be necessary.
18:29:34 <jaypipes> clarkb: and I believe it likely would... but I'm not 100% sure.
18:29:37 <trinaths> i have 2 hosts  [1] master Jenkins node [2] Slave jenkins node
18:29:40 <fungi> trinaths: upstream openstack infra uses something called nodepool to create and delete jenkins slaves for running jobs, so that each job runs on a pristine system... http://git.openstack.org/cgit/openstack-infra/nodepool/tree/README.rst
18:29:44 <sdague> jaypipes: unstack.sh would probably not be sufficient
18:29:54 <jaypipes> sdague: full DB wipe, etc?
18:30:07 <sdague> things like rabbit get grumpy over time
18:30:11 <jaypipes> ya
18:30:11 <anteaya> luqas: https://etherpad.openstack.org/p/third-party-ci-workshop
18:30:22 <sdague> clean.sh is an attempt at a full reset
18:30:31 <sdague> however, I wouldn't trust it for this use
18:30:48 <luqas> anteaya: thnx
18:30:54 <sdague> and realize the whole devstack-gate model is based on single use environments
18:30:57 <jaypipes> so what we need to do is either explain how to use nodepool properly (which does require significantly more compute resources for the third party), or describe how to reuse slave devstack nodes properly.
18:30:57 <anteaya> luqas: np, glad you are here
18:31:17 <sdague> because you are fundamentally downloading arbitrary code from the internet
18:31:31 <sdague> so the state it will leave the machine in, is very unknown
18:32:03 <anteaya> I always thought our testing model was based on single use only testing
18:32:09 <trinaths> jaypipes: as per the article-1&2, we have nodepool too installed in master node ...?
18:32:34 <fungi> to expand on sdague's point, i could propose a patch which does 'sudo rm -rf /usr/bin' ano no amount of unstack.sh fixing is going to undo that
18:32:40 <jaypipes> trinaths: it may be installed, but it is not used in the article. We manually create a devstcak slave node in the articles.
18:32:42 <trinaths> Jaypipes: can you guide me (now or from the updates on articles) on how this cleanup be achieved from nodepool ..
18:32:58 <trinaths> oh ! okay..
18:33:03 <jaypipes> trinaths: I could update the articles to use nodepool, but we would need a lot more compute resources at each vendor to do that...
18:33:04 <trinaths> interesting
18:33:43 <trinaths> jaypipes: okay... any simpler means to do this cleanup  ??
18:33:50 <roeyc> I've found amotoki devstack-clean.sh script to be helpful
18:33:56 <akerr> +1 for nodepool update to articles
18:33:57 <roeyc> https://github.com/amotoki/devstack-tools
18:34:01 <jaypipes> it would be great to find a way to re-use these slaves, but if there isnt a good way, I suppose I can update the articles to explain how to usse nodepool to manage a static set of VMs for slaves..
18:34:19 <jeblair> you might be able to do something with lvm snapshots.  we did that for a while on some bare-metal nodes.
18:34:45 <jaypipes> akerr: sure, I can work on that... (just be aware it's not free :) it does require additional compute resources...
18:35:03 <jaypipes> because you need a pool of available nodes (devstcak VM slaves) to run new jobs on.
18:35:10 <jaypipes> thus the name "nodepool" ;)
18:35:20 <jaypipes> but regardless, I will work on an updated article, no probs.
18:35:34 <trinaths> jaypipes: any other way other than nodepool
18:35:46 <jaypipes> in the meantime, perhaps trying out amotoki's devscatk-clean.sh is a good idea.
18:36:00 * jaypipes notes this question wasn't as easy as he thought it would be ... :)
18:36:01 <trinaths> i just have 2 hosts with 4GB RAM and 40GB HDD
18:36:15 <fungi> jaypipes: trinaths: the summary there is that since you're running code anyone can propose, and with root privileges, there's really no way to be absolutely certain the machine will be usable and uncompromised once the job finishes
18:36:17 <jaypipes> trinaths: yes, that is a problem...
18:36:34 <jaypipes> fungi: yes.
18:36:36 <jaypipes> understood.
18:36:38 <anteaya> jaypipes: trying to standardize everyone's unique system
18:36:43 <anteaya> hard
18:36:46 <jaypipes> indeed :)
18:36:49 <fungi> so you can pretend, but you're likely to be chasing occasional patches which broke your slaves
18:37:15 <jaypipes> OK, trinaths, you can try the devstcak-clean.sh approach for now, and I will work on documentation for nodepool setup in the meantime.
18:37:40 <jaypipes> trinaths: you will want to run the devscatck-clean.sh script before you run the devsctak-gate-vm-wrap.sh script in the jenksin job config..
18:37:53 <jaypipes> damn, I can never type devstack properly.
18:37:54 <trinaths> jaypipes: I will use this script in the post build actions of jenkins jobs
18:38:07 <jaypipes> trinaths: ok.
18:38:25 <trinaths> that way I get the clean env for next job
18:38:36 <jaypipes> ++
18:38:43 <jaypipes> trinaths: try it, at least.
18:38:50 <jaypipes> trinaths: and post to the ML with your findings, ok?
18:39:02 <jaypipes> alright, moving on to next question. Here we go:
18:39:03 <jaypipes> [3] For some tempest test cases, I get the message "Neutron skipped" but in the devstack logs, neutron is installed from git. Do I need to configure something more for neutron.
18:39:14 <trinaths> jaypipes: Sure..! will never let info hiding with me..
18:39:53 <jaypipes> Answer to #3:
18:39:54 <jaypipes> https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L151
18:40:06 <jaypipes> set the DEVSTACK_GATE_NEUTRON environ variaable to 1.
18:40:49 <trinaths> jaypipes: its again evnronment var .. export DEVSTACK_GATE_NEUTRON=1
18:40:53 <anteaya> trinaths: when you post use the tag [3rd party testing] in the subject line, and please post to the openstack development mailing list openstack-dev@lists.openstack.org
18:41:01 <jaypipes> trinaths: yup.
18:41:08 <trinaths> okay got it...
18:41:20 <trinaths> please go ahead to the next question.. kindly..
18:41:21 <jaypipes> OK, I've added the answer to the ehterpad on that question. All ready for next quesion?
18:41:23 <jaypipes> :)
18:41:30 <trinaths> :)
18:41:38 <jaypipes> [4] For services like Neutron, do we need to test cinder, nova, heat, swift etc.. too ??
18:41:46 <jaypipes> This is similar answer to #3.
18:41:52 <trinaths> jaypipes: sure will post with subject line [..] to openstack-dev list
18:41:55 <jaypipes> There are environ variables for all of those services.
18:42:17 <jaypipes> Easiest way to find these variables is to look in the devstack-gate-vm-wrap.sh script:
18:42:18 <trinaths> Huh! Thats really helpful..
18:42:26 <jaypipes> https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
18:42:29 <trinaths> I was struck on how to do it..
18:42:36 <trinaths> its a simple job as I see now..
18:42:38 <jaypipes> no worries. it's a maze of code to look through :)
18:42:49 <trinaths> :)
18:43:44 <trinaths> ona last wuestion haven't post in ML, hope many guys like me looking into it.. Jaypies: you artile-2 left-over exercise.. can u help on that "Publishing logs"
18:44:33 <anteaya> well we have spend 45 minutes on trinaths
18:44:44 <anteaya> it would be nice to move to someone else's question
18:44:52 <anteaya> I know luqas needs some time to talk
18:45:03 <jaypipes> trinaths: will do later. moving on to other questions.
18:45:07 <trinaths> please go ahead.
18:45:27 <trinaths> thanking all for helping me to find the answers to the questions... thanking all
18:45:29 <luqas> ok, regarding the rubbish left by a tempest failure
18:45:58 <luqas> broke our cloud in cloud
18:46:04 <luqas> need to prevent that again
18:46:29 <anteaya> jaypipes: what question did you want to address?
18:46:57 <jaypipes> anteaya: there was a general question from lukego above about which HOWTO to follow...
18:47:01 <jaypipes> real quick.
18:47:02 <anteaya> I would like to address luqas's concerns but don't want to step on your toes
18:47:05 <luqas> is there a safe way to run tempest preventing to leave rubbish in a failure?
18:47:56 <jaypipes> lukego: answer to that is that it's up to you. However, I will point out that my series of article and the os-ext-testing repo are built using as much of the upstream infra team's puppet modules as possible, and with much help from the infra team.
18:48:05 <anteaya> luqas: a failure will leave a stacktrace
18:48:13 <anteaya> luqas: if that is what you mean by rubbish
18:48:22 <trinaths> tempest failure rubbish.. u mean logs, services running.. tempest data in the DB.
18:48:35 <luqas> yes,
18:48:40 <jaypipes> lukego: so the os-ext-testing repo will likely be a good bet for you, but of course, there's bugs in it and you will need patience :)
18:49:26 <jaypipes> the next question in order was from VinayMahuli:
18:49:28 <jaypipes> To access gerrit from another host in a network we need to put gerrit.war web app in a web server called Jetty or tomcat.       I tried to integrate gerrit.war with tomcat, but it didn't work for me (on tomcat, the web app gerrit.war got deployed, but wasn't able to access it from the browser). Can you give some pointers on deploying gerrit.war in a tomcat webserver?
18:49:46 <VinayMahuli> yes
18:49:55 <luqas> rest of topologies
18:50:08 <lukego> jaypipes: OK. So, we were one of the first to dive into setting up 3rd party testing, and despite good intentions we failed spectacularly. I'd quite like to now take whatever the most conservative route to being compliant is. Does that mean I should wait for some bugs to be shaken out first? (when's a good time to set it up?)
18:50:20 <jaypipes> VinayMahuli: I would recommend using the Puppet modules from the upstream infra team. They are tried and true, and they work. The os-ext-testing repo uses those puppet modules to install Jenkins.
18:50:43 <anteaya> ah sorry I was mistaking luqas for lukego
18:51:11 <VinayMahuli> can you please give the reference of a weblink which you are talking about?
18:51:12 <jaypipes> lukego: perhaps you might wait one week or so until the nodepool updates are done, and then follow my article series?
18:51:20 <jaypipes> VinayMahuli: sure, one se.
18:51:22 <jaypipes> sec
18:51:31 <anteaya> lukego: your system has the ability to vote on changes to the sandbox repo: https://review.openstack.org/#/dashboard/9695
18:51:41 <anteaya> lukego: I haven't seen any activity
18:51:53 <jaypipes> VinayMahuli: here you go:
18:51:57 <jaypipes> #link http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/
18:52:03 <jaypipes> #link http://www.joinfu.com/2014/02/setting-up-an-openstack-external-testing-system-part-2/
18:52:13 <anteaya> lukego: also the last post to the mailing list had markmcclain state he wanted to see stability from your system in the sandbox first
18:52:27 <jaypipes> VinayMahuli: those walk through setting up stuff using the os-ext-testing repo, here:
18:52:45 <lukego> anteaya: How do I switch to the sandbox repo? (I was looking for that in Jenkins config today)
18:52:51 <jaypipes> #link https://github.com/jaypipes/os-ext-testing
18:52:51 <VinayMahuli> ok
18:53:02 <lukego> anteaya: The sandbox repo I found has only two files or so in it. Is that the right one?
18:53:25 <jaypipes> VinayMahuli: all that repo is is a thin layer of scripts that use the upstream Puppet modules from here:
18:53:27 <jaypipes> #link https://github.com/openstack-infra/config/
18:53:38 <anteaya> lukego: openstack-dev/sandbox is the name of the repo: http://git.openstack.org/cgit/openstack-dev/sandbox/
18:53:39 <trinaths> anteaya: sandbox repo has only 1 files.. test.py
18:54:00 <lukego> anteaya: also - we did our own integration test essentially by running the unit tests but with the plugin enabled and connected to a live external system. Is that enough for compliance? or do we need to switch to tempest?
18:54:00 <VinayMahuli> thank you
18:54:08 <anteaya> jaypipes: when you are done, can you address how to use the sandbox repo?
18:54:28 <VinayMahuli> I will go through the links
18:54:38 <jaypipes> anteaya: sure, pls add that to the etherpad?
18:54:44 <anteaya> yes
18:54:45 <jaypipes> next question was from akerr
18:54:47 <jaypipes> jaypipes: question �  this may be specific to cinder certification, or maybe apply to other projects as well:  Is the idea of 3rd party testing to only test the "common" community tempest tests and report that those work; or should we run against the community tempest tests as well as our own internally customized tests (that exercise vendor specific functionality) and report +/- on those combined results?
18:56:16 <jaypipes> akerr: great question. the idea behind third party testing is at least twofold (and irrespective of neutron vs. cinder vs. nova, etc): 1) to increase the testing surface and footprint for vendor drivers in the codebases and 2) to ensure that changes from one vendor or contributor don't inadvertantly break another vendor's code.
18:57:14 <jaypipes> akerr: as such, the tests that will be executed for a project like cinder at a third party location should be a set of community-agreed-upon Tempest tests that stress the code coverage for the public API that the vendor's driver implements.
18:57:17 <trinaths> jaypipes: true, this is what I learned form ML2 plugin meeting
18:57:52 <jaypipes> akerr: in addition to this baseline set of Tempest tests, third party CI systems are free to execute any number of additional tests against their driver code and integration between other components.
18:57:52 <akerr> jaypipes: so vendor specific tests would not be considered gating?
18:58:11 <krtaylor> unit vs. gate
18:58:31 <jaypipes> akerr: likely no, they would not. but that does not mean that vendor-specific tests (internal vendor implementation stuffs) could not or should not be run and exposed in the review comments.
18:58:46 <jaypipes> akerr: those tests would just likely be "NON-VOTING" tests.
18:59:02 <jaypipes> akerr: but would still definitely appear on the code review as a warning to the contributor.
18:59:06 <akerr> jaypipes: makes sense
18:59:14 <fungi> openstack infrastructure does not actively prevent changes which fail third-party tests from merging, but the core reviewers on those projects very well may take those results into consideration if the reporting systems are deemed reliable
18:59:26 <jaypipes> akerr: over time, who knows, those vendor-specific tests may be voting... it would depend on the PTL and the community's decision I suppose.
19:00:10 <anteaya> one minute
19:00:16 <fungi> probably better not to set that expectation. the infrastructure team is fairly strongly against the idea of having the openstack project's development able to be broken by outside organizations systems
19:00:17 <anteaya> oh out of time
19:00:45 <jaypipes> OK, luqas, unfortunately I did not gett to you in the time limit. We can move to #openstack-dev or -neutron to discuss if you would like?
19:00:45 <anteaya> jaypipes: how do you want to address remaining questions jaypipes?
19:00:49 <jaypipes> #endmeeting