18:02:18 <jaypipes> #startmeeting third-party-ci
18:02:20 <openstack> Meeting started Mon Mar 17 18:02:18 2014 UTC and is due to finish in 60 minutes.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:24 <openstack> The meeting name has been set to 'third_party_ci'
18:02:32 <jaypipes> good afternoon folks.
18:02:32 <jgriffith> hey ya
18:02:39 <luqas> hi
18:02:43 <jaypipes> #link https://etherpad.openstack.org/p/third-party-ci-workshop
18:02:48 <krtaylor> hello all
18:03:02 <jaypipes> I spent some time cleaning up the above etherpad... let's continue to use it.
18:04:03 <jaypipes> For question #2, mtreinish, jeblair and clarkb have gotten a DEVSTACK_TEMPEST_REGEX env variable into devstack-gate now.
18:04:23 <jaypipes> so I should update that section accordingly. IIRC, lukego has experimented successfully with it.
18:05:06 <jaypipes> If you're on the etherpad, please do make sure you set your name in the right-hand pane please (click on the little person icon thingee and enter you name)
18:08:51 <jaypipes> OK, sorry about that. Done cleaning up question 2.
18:09:04 <jaypipes> Does anyone have questions that we can tackle today on third party CI?
18:09:16 <jaypipes> anyone have any updates about how their CI installation/setup is going?
18:09:38 <jaypipes> jgriffith: how is Solidfire's going? run into roadbumps we can assist with?
18:10:17 <jgriffith> jaypipes: yes please
18:10:22 <jgriffith> jaypipes: I'm reproducing the issue now
18:10:31 <jgriffith> jaypipes: ran into problems just setting up :(
18:10:32 <jaypipes> jgriffith: ok, sounds good.
18:10:44 <jaypipes> jgriffith: perfect, that's what this meeting is all about :)
18:10:53 <jgriffith> Coolness
18:11:27 <jgriffith> jaypipes: so I spin up a simple 12.04 instance, copy my keys over
18:11:39 <jgriffith> clone my repo and wget the script
18:11:51 <jgriffith> The problem I've been hitting is that I get a couple of errors
18:12:02 * jgriffith can't find the pastebin so running again
18:12:07 <jaypipes> no probs
18:13:21 <rfolco> jaypipes, very quick question to confirm... "recheck <3rd-party>" is triggering Jenkins too. recheck-<3rd-party> with dash will do the trick ?
18:14:25 <jaypipes> rfolco: hmm, I don't think so, but lemme check the regex...
18:14:55 <jaypipes> rfolco: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/zuul/layout.yaml#L15
18:15:10 <jaypipes> rfolco: I do not believe a dash will work, no.
18:15:33 <rfolco> jaypipes, we did this:         comment_filter: (?i)^\s*recheck myvendor\s*$
18:15:41 <jaypipes> rfolco: if you really wanted (not recommended because it diverges from upstream), you could change that regex if you wish
18:16:00 <jaypipes> rfolco: that's a different specific regex, here: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/zuul/layout.yaml#L20
18:16:11 <jaypipes> rfolco: you are welcome to change that to whatever you want.
18:16:42 <jaypipes> rfolco: the first regex, though, is what the upstream system typically uses for recheck bug notifiers.
18:17:02 <jaypipes> rfolco: the second regex is something you can change to have a custom "only recheck on my system" event.
18:17:14 <jaypipes> rfolco: make sense?
18:17:28 <rfolco> jaypipes, yes. We actually added the second one.
18:17:30 <jaypipes> kk
18:17:43 <rfolco> jaypipes, not sure why its triggering Jenkins too.
18:18:27 <jaypipes> rfolco: are you asking why recheck-$your_vendor is triggering a jenkins job run?
18:19:03 <rfolco> jaypipes, I was asking if a dash is necessary to run our CI ONLY.
18:19:09 <rfolco> example: https://review.openstack.org/#/c/79239/
18:19:13 <jaypipes> no, it should not be at all.
18:20:06 <jaypipes> rfolco: no, no dash should be necessary if your regex is (?i)^\s*recheck pkvm\s*$
18:20:14 <rfolco> jaypipes, it has triggered Jenkins too.... I'll double check what happened. Thank you for confirming.
18:20:18 <jgriffith> jaypipes: FWIW I managed to get passed the issue I was seeing before, see if I make it through everything else :)
18:20:33 <jaypipes> jgriffith: do you remember what the old issue was?
18:20:44 <jgriffith> jaypipes: I'll find it
18:20:49 <jaypipes> k
18:22:32 <jgriffith> jaypipes: http://paste.openstack.org/show/73667/
18:23:09 <jgriffith> jaypipes: uncertain why the httpd stuff was having issues
18:23:33 <jaypipes> jgriffith: yeah, that last line meant Jenkins was still starting by the time jenkins-jobs was being executed.
18:23:41 <jgriffith> Oh interesting
18:23:47 <jaypipes> jgriffith: nothing to worry about. might just have been a slow startup initially.
18:24:10 <jaypipes> either a really fast puppet or a really slow Jenkins :)
18:24:29 <jgriffith> jaypipes: so timing is everything :)
18:24:55 <jaypipes> jgriffith: well, sometimes services say they are up (jenkins has a nasty habit of this), but they aren't fully up.
18:25:12 <jgriffith> ahh... that's rather annoying
18:25:36 <jaypipes> jgriffith: for example, you can shut down jenkins (service jenkins stop), then wait a second, then do service jenkins start, and then immediately run jenkins-jobs, and it will fail.
18:25:51 <jgriffith> jaypipes: oh, nice
18:25:55 <jaypipes> jgriffith: because jenkins service reported success to init.d, but it's not fully uip.
18:26:09 <jgriffith> jaypipes: I'll def want to keep that in mind I think
18:26:26 <jaypipes> yeah... especially if you do restarts of jenkins during debugging of the CI env.
18:26:29 <jaypipes> something to keep in mind.
18:28:13 <jgriffith> jaypipes: oh crap
18:28:23 <jgriffith> jaypipes: that was slave (part 2)
18:28:26 <jaypipes> always like to hear that ...
18:28:34 <jgriffith> jaypipes: the issues I was seeing were on master (part 1)
18:28:43 <jaypipes> jgriffith: slave shouldn't install jenkins-jobs, jenkins, or zuul... :(
18:28:56 <jaypipes> jgriffith: did you by accident run install_master.sh on the slave VM? :)
18:29:16 <jgriffith> jaypipes: so initially I was just doing the master
18:29:30 <jgriffith> jaypipes: I spun up a fresh instance to try again but accidentally ran slave setup
18:29:57 <jgriffith> jaypipes: I'll need to build another instance for the master which is where I saw that original error
18:30:02 <jgriffith> jaypipes: make sense?
18:30:15 <jaypipes> jgriffith: I see :) yeah, you'll need to redo the master I'm afraid.
18:30:37 <jaypipes> jgriffith: and if you changed your jenkins SSH keys, you will need to redo the slave as well.
18:30:58 <jgriffith> jaypipes: k
18:32:16 <jaypipes> luqas: how is your CI going? any issues we can assist with?
18:33:54 <luqas> jaypipes: hi
18:34:15 <luqas> well continous work on our CI system
18:34:23 <luqas> always something to do!
18:34:29 <jaypipes> :) indeed.
18:35:22 <luqas> I have a couple of things done before I get a concrete question, so I'll wait for next week I guess
18:35:55 <jaypipes> ok, no worries, luqas. you can also email the openstack-infra ML with questions and I will gather for next week's meeting, too.
18:36:20 <luqas> jaypipes: perfect
18:37:53 <krtaylor> Just fyi, as I have discussed a few times in -infra, I am organizing a 3rd party implementer self-help group to try and offload the infra core folks a little
18:38:26 <jaypipes> krtaylor: cool, do you want to that group to take over this meeting?
18:38:36 <krtaylor> I created an etherpad to get some ideas down #link https://etherpad.openstack.org/p/3rdPartyTesting
18:39:02 <krtaylor> jaypipes, well, I think initially this meeting could be a great input to that
18:39:09 <jaypipes> krtaylor: could you link to the etherpad with the Q&A?
18:39:18 <krtaylor> jaypipes, will do
18:39:43 <jaypipes> cheerio. yes, krtaylor would be great to align efforts and have a single place for 3rd party implementers to meet and help each other.
18:40:07 <krtaylor> one of the first things I'd like to do is standardize the ML tag, mainly because my mail filter is horrible
18:40:19 <jaypipes> hehe, no crap. agreed ;)
18:40:45 <jaypipes> krtaylor: how about 3PCI?
18:41:24 <krtaylor> hm, I'll add that, may be too short for beginners though
18:41:35 <jaypipes> possibly, yeah.
18:42:26 <jaypipes> krtaylor: so, my long term goal for all of this was to push pieces of my blog, the os-ext-testing and os-ext-testing-data repos, and all of the Q&A material into ci.openstack.org/third-party-testing
18:42:40 <jaypipes> krtaylor: do you concur with that long-term goal?
18:43:21 <jaypipes> krtaylor: I've actually been delaying doing any patches to the ci documentation upstream while the os-ext-testing stuff bakes a bit (and even will propose os-ext-testing into a stackforge repo when it's fully baked)
18:43:30 <krtaylor> yes, I see a lot of the same questions over and over internally, the Q&A distilled to a FAQ and links would be a huge help
18:43:37 <jaypipes> great
18:44:15 <jaypipes> well, how about we make it a goal to, by atlanta, have made good progress on that?
18:44:23 <krtaylor> agreed, this may wind up being a set of doc patches when all done
18:44:24 <krtaylor> yes
18:44:34 <jaypipes> getting all the docs up into ci.openstack.org, and having os-ext-testing turned into a stackforge repo?
18:45:53 <jgriffith> jaypipes: so not sure how to debug this... the master script finished
18:46:07 <jgriffith> jaypipes: but can't connect to the jenkins web interface (times out)
18:46:18 <jaypipes> jgriffith: k, can you see jenkins GUI at $HOST_IP:8080?
18:46:23 <krtaylor> jaypipes, I think that's a good goal
18:46:43 <jgriffith> jaypipes: no, that's what I mean when I say connection times out
18:46:51 <jgriffith> jaypipes: ps -ef shows jenkins running
18:47:00 <jgriffith> but it seems the web portion is still jacked up
18:47:31 <jaypipes> jgriffith: tail -n50 /var/log/jenkins/*.log?
18:47:37 <jgriffith> jaypipes: just did
18:47:41 <jaypipes> jgriffith: to a pastebin...
18:47:43 <jgriffith> says "fully up and running"
18:48:10 <jaypipes> hmm, odd... is there an iptables rule on your main host that is preventing traffic to the master's IP?
18:48:13 <jgriffith> jaypipes: http://paste.openstack.org/show/73685/
18:48:24 <jgriffith> jaypipes: ahh
18:48:27 <jgriffith> jaypipes: bet that's it
18:48:38 <jgriffith> jaypipes: actually I'll be it's my openstack secgroups
18:49:02 <jaypipes> yup.
18:49:15 <jaypipes> I've not seen this before either: Dependency token-macro (1.5.1) doesn't exist
18:49:21 <jaypipes> not sure if that is important or not, though.
18:49:28 <jgriffith> me neither :)
18:49:38 <jaypipes> clarkb, fungi: ever seen "Dependency token-macro (1.5.1) doesn't exist" in a jenkins log before?
18:49:41 <jgriffith> yeah, so I've only got 22
18:49:47 <jgriffith> on TCP
18:49:51 <jaypipes> jgriffith: open that puppy up!
18:49:53 <jaypipes> :P
18:49:59 <jaypipes> 8080
18:50:28 <clarkb> jaypipes: I have not
18:50:39 <clarkb> but guessing its an interplugin dependency that needs resolving
18:50:52 <jgriffith> jaypipes: yay for me
18:50:58 <jaypipes> :)
18:50:59 <jgriffith> that was it
18:51:19 <jaypipes> clarkb: prolly, yeah... let's see if opening up the sec groups helped that.
18:51:25 <jgriffith> jaypipes: any other ports off the top of your head while I'm at it?
18:51:26 <jaypipes> jgriffith: re-run install_master.sh
18:51:40 <jgriffith> jaypipes: do I need to clean anything out manually first?
18:51:44 <jgriffith> jaypipes: or just rerun?
18:51:44 <jaypipes> jgriffith: no
18:51:45 <fungi> jaypipes: not familiar looking to me
18:51:48 <jgriffith> k
18:52:07 <jaypipes> jgriffith: just 80, 8080, 443 and 22
18:53:30 <fungi> also, late in responding, but i'm very much in favor of krtaylor's proposal to lighten the load on infra, as long as that group is not afraid to ask us questions (please don't disappear and start coming up with solutions in a vacuum!)
18:54:29 <krtaylor> fungi, agreed, we'll have to keep you all in the loop somehow
18:54:42 <jaypipes> fungi: ++
18:54:47 <jgriffith> jaypipes: reran, don't see any errors in the new run
18:54:51 <jgriffith> jaypipes: think I'll proceed
18:55:00 <jaypipes> coo
18:55:58 <fungi> always happy to help when there are questions, but an faq would be a great start (something deeper than the current third-party ci doc we have--it desperately wants to be improved)
18:57:17 <jaypipes> gonna end this meeting since infra is up next for their weekly meeting... see you all next week.
18:57:20 <jaypipes> #endmeeting