18:01:06 #startmeeting third-party-ci 18:01:07 Meeting started Mon Mar 3 18:01:06 2014 UTC and is due to finish in 60 minutes. The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:10 The meeting name has been set to 'third_party_ci' 18:01:32 Welcome all. 18:01:37 I will be taking some notes on this etherpad: 18:01:38 #link https://etherpad.openstack.org/p/third-party-ci-workshop 18:01:54 as we answer questions, we will put the question and answer on that etherpad. 18:02:17 I'd like to ask up front that we focus on answering *one* question at a time, please. 18:02:32 for folks who haven't used etherpad before, if you clirk the "person" icon in the top right, you can enter your name, then when you make edits, we can see who edited what 18:02:44 'click', not 'clirk' 18:02:51 I'm going to go in order, with trinaths's questions, then roeyc, then VinayMahuli . 18:03:02 * jaypipes clirks on jeblair 18:03:11 * jeblair makes a clirking sound 18:03:39 OK. Here's the first question, from Trinath's ML post: 18:03:41 [1] sandbox-dvsm-tempest-full runs all the test cases where few fail, causing the completed build to fail. 18:03:41 How to restrict the testcases? 18:04:01 okay 18:04:48 trinaths: OK, first things first, let's please refer to this as just "tempest-dsvm-full" (as it has been renamed in the upstream repository to better reflect that the job is not restricted to the sandbox project) 18:04:48 You can edit the tox.ini from /opt/stack/new/tempest/ to add tests you want to skip 18:05:29 ArxCruz:one sec... 18:05:36 jaypipes: :-# 18:05:47 #link https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/examples.yaml#L11 18:06:02 s/tempest-dsvm-full/ddsvm-tempest-full/ ... sorry 18:06:32 jaypipes: okay.. let me search the same 18:06:38 trinaths: so, Tempest can run any number of subsets of tests 18:07:33 trinaths: what you need to find out is how to indicate to tempest which of these tests get run, and do this using Bash scripts in your Jenkns jobs configuration. So, let's show how to do that. 18:07:37 * jaypipes grabs a link... 18:08:02 jaypipes: one Question here: since tempest-full is meant to run any number or test cases, then how do we restrict to few.. like skip cinder and swift like so.. 18:08:12 jaypipes: okay please go ahead.. 18:08:25 trinaths, all: OK, please go here: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L326 18:08:27 #link https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L326 18:09:09 This is the script that runs the devstcak-based tests in the gate, and from that line and below, you will see different *invocations* of running tempest *using the tox CLI tool* 18:09:21 sdague: ^ do we need a devstack-gate parameter to make it easier to configure what tests tempest runs? 18:09:39 sdague: i think some stackforge projects could use that too.... 18:10:00 (this might be off-topic, we can take it into #openstack-infra) 18:10:05 jeblair: it'll probably be helpful 18:10:08 jeblair: that would make things easier/more flexible, but looking at the devstack-gate script is a good place to see how everything works (for folks running their own scripts..) 18:10:23 the tox all job can take a regex but we don't have a way to do that with devstack-gate 18:10:37 jaypipes: yeah, i was thinking that 3rd party folks might be able to reuse devstack-gate more directly if we did that 18:10:49 trinaths: so, as an example, having the DEVSTACK_GATE_TEMPEST_TESTR_FULL env variable set to 1, would execute this little guy: 18:10:49 sudo -H -u tempest tox -etestr-full -- --concurrency=$TEMPEST_CONCURRENCY 18:11:04 mtreinish: jeblair jaypipes what we are doing in our side, is switch the tox.ini adding the jobs we want to skip 18:11:52 ArxCruz: that is also a perfectly fine method of configuring tempest. I presume you are doing some sort of sed/awk magic to add/replace things in the tox.ini file? 18:12:09 jaypipes: true.. ! i understand.. we need to configure them as 'export' vars .. for automated jenkins jobs 18:12:24 jaypipes: we configure our jobs to copy our tox.ini from our git repo before it runs 18:12:34 for example, in [testenv:full] section of tox.ini we have this: 18:12:42 bash tools/pretty_tox.sh '(?!.*(test_volume_backup_create_get_restore_delete|\[.*\bslow\b.*\]).*)(^tempest\.(api|scenario|thirdparty|cli)) {posargs}' 18:12:53 this will skip the test_volume_backup_create_get_restore_delete test 18:13:02 trinaths: right. all of the DEVSTACK_GATE_TEMPEST_XXX variables can be set in your Jenkins job configuration for all of those individual blocks starting at line 326. 18:13:19 ArxCruz: can we teach those attending good irc habits by using paste.openstack.org? they may not know how 18:13:22 trinaths: and jeblair and sdague are discussing adding more of those types of variables for third-party testing uses. 18:13:37 anteaya: oh, sure 18:13:41 thx 18:13:49 * jaypipes making a couple notes in etherpad... 18:14:10 jaypipes: here's our tox.ini http://paste.openstack.org/show/71835/ 18:14:21 ArxCruz: :D 18:14:28 we are skipping some jobs that's unstable in our environment 18:14:46 ArxCruz: I would avoid doing that in the tempest tox.ini because that file changes somewhat regularly. So you may hit an issue because of that eventually. 18:15:43 jaypipes: also, the script its self writes an localrc file with these parameters. Can we modify .localrc.auto file for these required changes .. Any comments ?? 18:15:46 mtreinish: what you suggest then ? 18:16:19 ArxCruz: the tox 'all job' can take a regex as a posarg and work as expected 18:16:44 so you can run tox -eall -- 'regex filter' and it should work the same 18:17:20 mtreinish: ++ would you mind adding that note to the etherpad? 18:17:41 jaypipes: in the last statement you mean to say, that from the browser while configuring the jenkins job we need to inject the env variables.. ?? 18:17:52 jaypipes: sure, I'm also drafting a devstack-gate patch per your suggestion to add that functionality to d-g 18:18:12 mtreinish: awesome, ty 18:18:32 trinaths: yes. note here: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/examples.yaml#L33 18:18:56 * jeblair stops drafting a devstack-gate patch 18:19:00 trinaths: that we export an environ variable. That would be where you would set the appropriate DEVSTACK_GATE_TEMPEST_XXX variable. 18:19:12 trinaths: acutally, can be more than one of those variables... 18:19:26 trinaths: for instance, if you only wanted to run the Tempest smoke tests, you would put: 18:19:53 export DEVSTACK_GATE_SMOKE_SERIAL=1 18:20:02 trinaths: ^^ for the serial-running smoke tests... 18:20:19 jaypipes: okay... got it.. 18:20:30 jaypipes: its clear 18:21:18 trinaths: and what mtreinish, jeblair and others having been talking about is adding more flexibility to that part of the devstcak-gate-vm.sh script to allow things like custom regexes, etc. 18:21:59 trinaths: OK to move on to your second question? 18:22:03 jaypipes: mtreinish so, the -efull also have this functionality ? or just -eall ? 18:22:03 jaypipes: okay.. I'm viewing the etherpad.. for the same.. that really helps me .. 18:22:19 To access gerrit from another host in a network we need to put gerrit.war web app in a web server called Jetty or tomcat. I tried to integrate gerrit.war with tomcat, but it didn't work for me (on tomcat, the web app gerrit.war got deployed, but wasn't able to access it from the browser). Can you give some pointers on deploying gerrit.war in a tomcat webserver? 18:22:38 "tox -eall -- 'regex filter'" where to specify this command. 18:22:50 ArxCruz: -efull and -eALL are different, IIRC. -eALL is just telling tox to execute the tests that are specified in the tox.ini section to represent "ALL tests" 18:23:07 I have just made a new heading on the etherpad for Questions 18:23:22 trinaths: I believe we will be adding a way to specify the regex in an environ variable. 18:23:28 those with questions please post them there and jaypipes will select them one at a time 18:23:40 VinayMahuli: can you do that? 18:23:46 sure 18:23:51 trinaths: so that all you would do is export DEVSTACK_GATE_TEMPEST_REGEX or something like that, in your Jenksin Job config. 18:24:08 jaypipes: question — this may be specific to cinder certification, or maybe apply to other projects as well: Is the idea of 3rd party testing to only test the "common" community tempest tests and report that those work; or should we run against the community tempest tests as well as our own internally customized tests (that exercise vendor specific functionality) and report +/- on those combined results? 18:24:10 trinaths: OK, I am moving on to your next question now. 18:24:15 jaypipes: ok 18:24:20 jaypipes: sure .. please go ahead 18:24:22 akerr: noted. will answer in order. 18:24:44 [2] After the build is failed, still services like nova, swift are running. Does Jenkins using devstack unstack the stack. 18:24:50 OK, this one is easy :) 18:25:37 trinaths: so, as you correctly note, these services remain running (as devstack controls the screen sessions in which those services are running) 18:26:31 jaypipes: Here, I want understand that, so these services affect the next patch set under test ? 18:26:39 trinaths: Jenkins does not stop those services, nor run unstack.sh, AFAIK. Though there may be upgrade (grenade) scenarios where that might happen. Could jeblair, sdague or dtroyer comment on that? 18:27:24 devstack-gate doesn't unstack 18:27:24 jaypipes: trinaths: we run all of our tests on single use slaves that are deleted after the tests complete. As a result the tests don't stop services 18:27:26 we assume you create a clean environment before it runs 18:27:28 and that the state at the end is throw away 18:27:41 when a old build fails, all the openstack services run as they are.. when a new patch arrives for test, devstack will start from 'git clone' ... till the end.. will this not affect the test. 18:27:59 trinaths: it won't because we never run tests on a machine more than once 18:28:22 clarkb: trinaths is asking for non-single-use slaves :) 18:28:31 okay.. but how do we clean the environment used for previous test 18:28:33 jaypipes: there is no such thing with upstream devstack testing 18:28:42 clarkb: understood. 18:28:44 jaypipes: true .. 18:29:03 anteaya: can you share the etherpad link please, I was late 18:29:10 clarkb: however, what trinaths is asking is for non-single-use slaves, whether an unstack.sh would be necessary. 18:29:34 clarkb: and I believe it likely would... but I'm not 100% sure. 18:29:37 i have 2 hosts [1] master Jenkins node [2] Slave jenkins node 18:29:40 trinaths: upstream openstack infra uses something called nodepool to create and delete jenkins slaves for running jobs, so that each job runs on a pristine system... http://git.openstack.org/cgit/openstack-infra/nodepool/tree/README.rst 18:29:44 jaypipes: unstack.sh would probably not be sufficient 18:29:54 sdague: full DB wipe, etc? 18:30:07 things like rabbit get grumpy over time 18:30:11 ya 18:30:11 luqas: https://etherpad.openstack.org/p/third-party-ci-workshop 18:30:22 clean.sh is an attempt at a full reset 18:30:31 however, I wouldn't trust it for this use 18:30:48 anteaya: thnx 18:30:54 and realize the whole devstack-gate model is based on single use environments 18:30:57 so what we need to do is either explain how to use nodepool properly (which does require significantly more compute resources for the third party), or describe how to reuse slave devstack nodes properly. 18:30:57 luqas: np, glad you are here 18:31:17 because you are fundamentally downloading arbitrary code from the internet 18:31:31 so the state it will leave the machine in, is very unknown 18:32:03 I always thought our testing model was based on single use only testing 18:32:09 jaypipes: as per the article-1&2, we have nodepool too installed in master node ...? 18:32:34 to expand on sdague's point, i could propose a patch which does 'sudo rm -rf /usr/bin' ano no amount of unstack.sh fixing is going to undo that 18:32:40 trinaths: it may be installed, but it is not used in the article. We manually create a devstcak slave node in the articles. 18:32:42 Jaypipes: can you guide me (now or from the updates on articles) on how this cleanup be achieved from nodepool .. 18:32:58 oh ! okay.. 18:33:03 trinaths: I could update the articles to use nodepool, but we would need a lot more compute resources at each vendor to do that... 18:33:04 interesting 18:33:43 jaypipes: okay... any simpler means to do this cleanup ?? 18:33:50 I've found amotoki devstack-clean.sh script to be helpful 18:33:56 +1 for nodepool update to articles 18:33:57 https://github.com/amotoki/devstack-tools 18:34:01 it would be great to find a way to re-use these slaves, but if there isnt a good way, I suppose I can update the articles to explain how to usse nodepool to manage a static set of VMs for slaves.. 18:34:19 you might be able to do something with lvm snapshots. we did that for a while on some bare-metal nodes. 18:34:45 akerr: sure, I can work on that... (just be aware it's not free :) it does require additional compute resources... 18:35:03 because you need a pool of available nodes (devstcak VM slaves) to run new jobs on. 18:35:10 thus the name "nodepool" ;) 18:35:20 but regardless, I will work on an updated article, no probs. 18:35:34 jaypipes: any other way other than nodepool 18:35:46 in the meantime, perhaps trying out amotoki's devscatk-clean.sh is a good idea. 18:36:00 * jaypipes notes this question wasn't as easy as he thought it would be ... :) 18:36:01 i just have 2 hosts with 4GB RAM and 40GB HDD 18:36:15 jaypipes: trinaths: the summary there is that since you're running code anyone can propose, and with root privileges, there's really no way to be absolutely certain the machine will be usable and uncompromised once the job finishes 18:36:17 trinaths: yes, that is a problem... 18:36:34 fungi: yes. 18:36:36 understood. 18:36:38 jaypipes: trying to standardize everyone's unique system 18:36:43 hard 18:36:46 indeed :) 18:36:49 so you can pretend, but you're likely to be chasing occasional patches which broke your slaves 18:37:15 OK, trinaths, you can try the devstcak-clean.sh approach for now, and I will work on documentation for nodepool setup in the meantime. 18:37:40 trinaths: you will want to run the devscatck-clean.sh script before you run the devsctak-gate-vm-wrap.sh script in the jenksin job config.. 18:37:53 damn, I can never type devstack properly. 18:37:54 jaypipes: I will use this script in the post build actions of jenkins jobs 18:38:07 trinaths: ok. 18:38:25 that way I get the clean env for next job 18:38:36 ++ 18:38:43 trinaths: try it, at least. 18:38:50 trinaths: and post to the ML with your findings, ok? 18:39:02 alright, moving on to next question. Here we go: 18:39:03 [3] For some tempest test cases, I get the message "Neutron skipped" but in the devstack logs, neutron is installed from git. Do I need to configure something more for neutron. 18:39:14 jaypipes: Sure..! will never let info hiding with me.. 18:39:53 Answer to #3: 18:39:54 https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L151 18:40:06 set the DEVSTACK_GATE_NEUTRON environ variaable to 1. 18:40:49 jaypipes: its again evnronment var .. export DEVSTACK_GATE_NEUTRON=1 18:40:53 trinaths: when you post use the tag [3rd party testing] in the subject line, and please post to the openstack development mailing list openstack-dev@lists.openstack.org 18:41:01 trinaths: yup. 18:41:08 okay got it... 18:41:20 please go ahead to the next question.. kindly.. 18:41:21 OK, I've added the answer to the ehterpad on that question. All ready for next quesion? 18:41:23 :) 18:41:30 :) 18:41:38 [4] For services like Neutron, do we need to test cinder, nova, heat, swift etc.. too ?? 18:41:46 This is similar answer to #3. 18:41:52 jaypipes: sure will post with subject line [..] to openstack-dev list 18:41:55 There are environ variables for all of those services. 18:42:17 Easiest way to find these variables is to look in the devstack-gate-vm-wrap.sh script: 18:42:18 Huh! Thats really helpful.. 18:42:26 https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh 18:42:29 I was struck on how to do it.. 18:42:36 its a simple job as I see now.. 18:42:38 no worries. it's a maze of code to look through :) 18:42:49 :) 18:43:44 ona last wuestion haven't post in ML, hope many guys like me looking into it.. Jaypies: you artile-2 left-over exercise.. can u help on that "Publishing logs" 18:44:33 well we have spend 45 minutes on trinaths 18:44:44 it would be nice to move to someone else's question 18:44:52 I know luqas needs some time to talk 18:45:03 trinaths: will do later. moving on to other questions. 18:45:07 please go ahead. 18:45:27 thanking all for helping me to find the answers to the questions... thanking all 18:45:29 ok, regarding the rubbish left by a tempest failure 18:45:58 broke our cloud in cloud 18:46:04 need to prevent that again 18:46:29 jaypipes: what question did you want to address? 18:46:57 anteaya: there was a general question from lukego above about which HOWTO to follow... 18:47:01 real quick. 18:47:02 I would like to address luqas's concerns but don't want to step on your toes 18:47:05 is there a safe way to run tempest preventing to leave rubbish in a failure? 18:47:56 lukego: answer to that is that it's up to you. However, I will point out that my series of article and the os-ext-testing repo are built using as much of the upstream infra team's puppet modules as possible, and with much help from the infra team. 18:48:05 luqas: a failure will leave a stacktrace 18:48:13 luqas: if that is what you mean by rubbish 18:48:22 tempest failure rubbish.. u mean logs, services running.. tempest data in the DB. 18:48:35 yes, 18:48:40 lukego: so the os-ext-testing repo will likely be a good bet for you, but of course, there's bugs in it and you will need patience :) 18:49:26 the next question in order was from VinayMahuli: 18:49:28 To access gerrit from another host in a network we need to put gerrit.war web app in a web server called Jetty or tomcat. I tried to integrate gerrit.war with tomcat, but it didn't work for me (on tomcat, the web app gerrit.war got deployed, but wasn't able to access it from the browser). Can you give some pointers on deploying gerrit.war in a tomcat webserver? 18:49:46 yes 18:49:55 rest of topologies 18:50:08 jaypipes: OK. So, we were one of the first to dive into setting up 3rd party testing, and despite good intentions we failed spectacularly. I'd quite like to now take whatever the most conservative route to being compliant is. Does that mean I should wait for some bugs to be shaken out first? (when's a good time to set it up?) 18:50:20 VinayMahuli: I would recommend using the Puppet modules from the upstream infra team. They are tried and true, and they work. The os-ext-testing repo uses those puppet modules to install Jenkins. 18:50:43 ah sorry I was mistaking luqas for lukego 18:51:11 can you please give the reference of a weblink which you are talking about? 18:51:12 lukego: perhaps you might wait one week or so until the nodepool updates are done, and then follow my article series? 18:51:20 VinayMahuli: sure, one se. 18:51:22 sec 18:51:31 lukego: your system has the ability to vote on changes to the sandbox repo: https://review.openstack.org/#/dashboard/9695 18:51:41 lukego: I haven't seen any activity 18:51:53 VinayMahuli: here you go: 18:51:57 #link http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ 18:52:03 #link http://www.joinfu.com/2014/02/setting-up-an-openstack-external-testing-system-part-2/ 18:52:13 lukego: also the last post to the mailing list had markmcclain state he wanted to see stability from your system in the sandbox first 18:52:27 VinayMahuli: those walk through setting up stuff using the os-ext-testing repo, here: 18:52:45 anteaya: How do I switch to the sandbox repo? (I was looking for that in Jenkins config today) 18:52:51 #link https://github.com/jaypipes/os-ext-testing 18:52:51 ok 18:53:02 anteaya: The sandbox repo I found has only two files or so in it. Is that the right one? 18:53:25 VinayMahuli: all that repo is is a thin layer of scripts that use the upstream Puppet modules from here: 18:53:27 #link https://github.com/openstack-infra/config/ 18:53:38 lukego: openstack-dev/sandbox is the name of the repo: http://git.openstack.org/cgit/openstack-dev/sandbox/ 18:53:39 anteaya: sandbox repo has only 1 files.. test.py 18:54:00 anteaya: also - we did our own integration test essentially by running the unit tests but with the plugin enabled and connected to a live external system. Is that enough for compliance? or do we need to switch to tempest? 18:54:00 thank you 18:54:08 jaypipes: when you are done, can you address how to use the sandbox repo? 18:54:28 I will go through the links 18:54:38 anteaya: sure, pls add that to the etherpad? 18:54:44 yes 18:54:45 next question was from akerr 18:54:47 jaypipes: question � this may be specific to cinder certification, or maybe apply to other projects as well: Is the idea of 3rd party testing to only test the "common" community tempest tests and report that those work; or should we run against the community tempest tests as well as our own internally customized tests (that exercise vendor specific functionality) and report +/- on those combined results? 18:56:16 akerr: great question. the idea behind third party testing is at least twofold (and irrespective of neutron vs. cinder vs. nova, etc): 1) to increase the testing surface and footprint for vendor drivers in the codebases and 2) to ensure that changes from one vendor or contributor don't inadvertantly break another vendor's code. 18:57:14 akerr: as such, the tests that will be executed for a project like cinder at a third party location should be a set of community-agreed-upon Tempest tests that stress the code coverage for the public API that the vendor's driver implements. 18:57:17 jaypipes: true, this is what I learned form ML2 plugin meeting 18:57:52 akerr: in addition to this baseline set of Tempest tests, third party CI systems are free to execute any number of additional tests against their driver code and integration between other components. 18:57:52 jaypipes: so vendor specific tests would not be considered gating? 18:58:11 unit vs. gate 18:58:31 akerr: likely no, they would not. but that does not mean that vendor-specific tests (internal vendor implementation stuffs) could not or should not be run and exposed in the review comments. 18:58:46 akerr: those tests would just likely be "NON-VOTING" tests. 18:59:02 akerr: but would still definitely appear on the code review as a warning to the contributor. 18:59:06 jaypipes: makes sense 18:59:14 openstack infrastructure does not actively prevent changes which fail third-party tests from merging, but the core reviewers on those projects very well may take those results into consideration if the reporting systems are deemed reliable 18:59:26 akerr: over time, who knows, those vendor-specific tests may be voting... it would depend on the PTL and the community's decision I suppose. 19:00:10 one minute 19:00:16 probably better not to set that expectation. the infrastructure team is fairly strongly against the idea of having the openstack project's development able to be broken by outside organizations systems 19:00:17 oh out of time 19:00:45 OK, luqas, unfortunately I did not gett to you in the time limit. We can move to #openstack-dev or -neutron to discuss if you would like? 19:00:45 jaypipes: how do you want to address remaining questions jaypipes? 19:00:49 #endmeeting