15:04:36 <anteaya> #startmeeting third-party
15:04:36 <openstack> Meeting started Mon Aug 24 15:04:36 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:04:39 <openstack> The meeting name has been set to 'third_party'
15:04:43 <akerr> o/
15:04:45 <asselin> o/
15:04:51 <anteaya> asselin: thanks for getting up early :)
15:04:52 <mmedvede> o/
15:04:56 <anteaya> how is everyone today?
15:05:41 <asselin> last week we had nodepool issues
15:05:43 <patrickeast> hey
15:05:56 <asselin> #link nodepool issues http://lists.openstack.org/pipermail/openstack-dev/2015-August/072556.html
15:06:35 <asselin> seems it's been hitting others as well, so read ^^ for solutions to get nodepool working again
15:06:43 <anteaya> so we would like to begin by discussing nodepool?
15:07:33 <mmedvede> we actually pin nodepool to a certain version to avoid such surprises
15:07:40 <anteaya> thanks for introducing the topic asselin
15:07:48 <asselin> mmedvede, how do you pin?
15:08:31 <asselin> but either way, still not sure why it broke....
15:09:05 <patrickeast> one thing kind of related, i have been having issues with my ci system killing vm's part-way through, in an effort to fix that i manually went and set my nodepool to v0.1.1 and zuul to v2.1.1 (at least i think those are the latest versions)
15:09:34 <patrickeast> i see that the nodepool and zuul puppet classes have a revision parameter, but i don't think we put that in the openstackci one
15:09:36 <patrickeast> was that intentional?
15:09:56 <patrickeast> i'm having much better results running on the older 'release' version instead of tip of tree
15:10:00 <mmedvede> asselin: I specify nodepool branch to use in my puppet manifest
15:10:45 <asselin> patrickeast, yea, should work on adding that
15:11:07 <asselin> well, using that
15:11:29 <patrickeast> asselin: ok cool, i can put up a patch to do it, just wanted to check and see if that was by design or not
15:11:58 <asselin> I wonder if nodepool/zuul have release tags that can be used
15:12:00 <anteaya> patrickeast: v0.1.1 for nodepool is a released tag, the latest tag for zuul is v2.1.0
15:12:13 <patrickeast> ahh yea 2.1.0 for zuul
15:12:23 <anteaya> so if you have v2.1.1 it is some commit after the latest tag
15:12:24 <patrickeast> anteaya: those are what i'm using, to much better effect than master
15:12:39 <anteaya> patrickeast: I'm glad you are having good results
15:13:00 <asselin> patrickeast, if you have time, put up a patch & i'll review
15:13:00 <mmedvede> There are a few useful commits after tagged commit, so I am using the commit sha as the revision
15:13:26 <anteaya> asselin: both nodepool and zuul have release tags, yes
15:13:50 <asselin> mmedvede, so which is a better default, latest release tag or master?
15:13:53 <patrickeast> does infra use the tagged versions?
15:14:04 <asselin> I beleive they use master
15:14:06 <anteaya> not that I am aware
15:14:49 <anteaya> yes I too believe we use master of all our tools
15:14:59 <patrickeast> maybe its worth figuring out kind of a stable recommended point for the 3rd party ci systems
15:15:07 <asselin> patrickeast, +1
15:15:16 <anteaya> well the problem with a recommendation is it gets old
15:15:26 <patrickeast> yea definitely
15:15:32 <anteaya> and many operators don't follow what we recommend anyway
15:15:37 <mmedvede> asselin: I normally use something that works, then manually review new commits, and switch to a new pin if it seems safe
15:15:59 <anteaya> I'm fine if by talking to each other your decide tag x and tag y are the flavour d'jour
15:16:17 <anteaya> but please don't communicate that further than the meeting logs
15:16:22 <anteaya> or irc conversation
15:16:35 <anteaya> as it just proves fodder for someone to drag their heels
15:16:53 <anteaya> "your recommendation didn't work for me, so I don't have to do what you say"
15:17:11 <patrickeast> i can understand that, but right now we recommend master... which also doesn't always work
15:17:13 <asselin> well, the good news is that there's a nodepool ci job that caught the issue and showed the fix solve it
15:17:24 <anteaya> sorry but unfortunately we are limited by the more relucant operators when trying to mix up standards
15:17:38 <anteaya> well master works for infra
15:17:51 <anteaya> and part of the point is to help upstream
15:18:04 <anteaya> so if it doesn't work, then it helps everyone to find out why and offer a patch
15:18:15 <anteaya> that is the point of open source work
15:18:22 <asselin> anteaya, actually in this case it worked for infra b/c nodepool wasn't restarted.
15:18:30 <anteaya> fair enough
15:18:36 <asselin> hence the urgency infra team had to help fix it
15:18:45 <anteaya> and yay for the nodepool ci job
15:18:48 <anteaya> yes
15:18:53 <asselin> +1
15:18:56 <anteaya> and that is the nature of the workflow
15:19:02 <anteaya> fraught with hazards
15:19:08 <anteaya> but we all have the same problems
15:19:21 <anteaya> and hopefully the same commitment to getting them working, for everyone
15:20:10 <anteaya> third party ci systems are proving valuable in that regard
15:20:23 <anteaya> they are sometimes finding issues before infra does
15:20:29 <anteaya> and that is very very helpful
15:20:57 <asselin> yeah, I guess our ci down saved the world. :)
15:20:59 <anteaya> for instance jgriffith found the openstack-client-config issue 2 weeks ago before anyone else did
15:21:08 <anteaya> asselin: it happens
15:21:14 <anteaya> and thanks for taking your turn
15:21:20 <anteaya> we really really appreciate it
15:21:52 * asselin will start making tshirts
15:21:58 <anteaya> so we were able to find a fix for the openstack-client-config issue and tag a new release prior to it hitting the gate and more third party ci systems
15:22:05 <anteaya> asselin: :)
15:22:18 <anteaya> so it is painful, yes it is
15:22:26 <anteaya> and frustrating, most certainly
15:22:40 <anteaya> but in the long run it helps everyone
15:22:45 <asselin> anteaya, what was the issue with openstack-client-config?
15:22:56 <anteaya> which for most folks in open source is the point of being in open source
15:23:06 * anteaya digs up the ml thread
15:24:06 <anteaya> #link http://lists.openstack.org/pipermail/openstack-dev/2015-August/071951.html
15:25:11 <asselin> thanks
15:25:17 <anteaya> thank you
15:25:31 <patrickeast> i'm a little bit confused, these are different issues the openstack-client-config thing was an issue with openstack ie the product we are supposed to be testing, not with nodepool/zuul/shade/etc, i can totally understand wanting to help test out and act as a canary for the infra team, but i have a dev-ci system for that... why should i allow my live system
15:25:31 <patrickeast> break and block patches in cinder? why should we say that is ok for all third party ci systems?
15:25:59 <patrickeast> imo there should still be a 'stable' point to work off of
15:26:21 <anteaya> I'm not stopping you from running any version of the tools you wish to run
15:26:29 <patrickeast> but i have to keep it a secret?
15:26:33 <anteaya> nor am I stopping you from discussing with others
15:26:47 <anteaya> I will not stand behind any version combination as recommended
15:26:53 <BobBall> I completely agree with patrickeast - I think as 3rd party CI operators we should run a common system in a known good configuration, including specific tags
15:27:02 <anteaya> you want to make recommendations as patrickeast, I can't stop you
15:27:15 <anteaya> BobBall: I am not stopping you
15:27:27 <anteaya> infra is not going to be issuing recommendations in that regard
15:27:34 <anteaya> other than running what we run
15:27:55 <mmedvede> Living on the tip normally means your CI system would be in almost constant state of fighting random fires
15:28:02 <anteaya> yes
15:28:06 <patrickeast> maybe then its a question for a different venue, but why isn't infra using tagged releases? wouldn't have help alieve issues with the gate too?
15:28:15 <anteaya> any the tests don't catch
15:28:15 <BobBall> So you're just saying  "Infra runs master, a group of 3rd party CI operators run according to doc over there -->"
15:28:21 <patrickeast> s/have/that/
15:28:38 <anteaya> I am saying that I am not making any more recommendations
15:28:48 <anteaya> as the ones we have now are not followed
15:29:12 <patrickeast> anteaya: understood
15:29:16 <anteaya> if you read the prohetstor ci and microsoft ci threads on the mailing list
15:29:34 <anteaya> personally I would have cut both loose long ago
15:29:44 <anteaya> but I support thingee's attempts to help them
15:29:56 <anteaya> and will not widen the gap to make his job tougher
15:30:00 <anteaya> I hear what you are saying
15:30:06 <anteaya> and I totally get it
15:30:13 <anteaya> you want your systems to work
15:30:18 <anteaya> a very good goal
15:30:28 <mmedvede> we want to help upstream too :)
15:30:29 <anteaya> I just can't choose your goal over the ptl's goal
15:30:41 <anteaya> which is to try to help those who wish to be included
15:30:45 <anteaya> mmedvede: thanks :)
15:30:47 <mmedvede> but we have limited time resources
15:30:51 <anteaya> I always hope for that as a goal
15:30:54 * anteaya nods
15:30:57 <anteaya> yes I get that too
15:31:03 <anteaya> loud and clear
15:31:37 <anteaya> so talking amoungst yourselves about best practices?
15:31:50 <anteaya> lovely I my heart applaudes every time I see it
15:31:58 <asselin> personally for our 3rd party ci, I don't mind to use master. However we also have internal ci systems we use for non-openstack, and plan to change those to tags/commits.
15:32:09 <anteaya> announcing to the world the one truth path? that at best has a stale date on it
15:32:23 <anteaya> and won't be in the group supporting that direction
15:32:29 <patrickeast> haha i still dont understand, thats the whole point of the converged openstack ci infrastructure isnt it?
15:32:43 <patrickeast> if we didn't want a 'true path' why are we doing that?
15:32:45 <asselin> I don't think recommending the latest tag is a bad recommendation
15:32:54 <anteaya> patrickeast: we are using the same tools yes
15:33:39 <BobBall> asselin: Can we get puppet to automatically determine the last tag, or do we have to provide a named tag?
15:34:08 <asselin> BobBall, good question. one way to do it would be to install for pypi
15:34:36 <anteaya> patrickeast: if we have everyone following the same menu, then updating the menu with the soup d'jour is easy
15:35:38 <patrickeast> anteaya: yea that makes perfect sense to me, what i don't get is the resistance to improve our current soups flavor in the mean-time just because not everyone is on the same menu
15:35:51 <anteaya> because not everyone is on the same menu
15:36:06 <patrickeast> so? they will be as left behind and screwed as they were yesterday
15:36:14 <patrickeast> or if they figure it out, more power to them
15:36:18 <anteaya> can you recognize the difference between an agreement at a meeting and making a recommendation to the mailing list
15:36:26 <patrickeast> yes
15:36:30 <anteaya> great
15:36:35 <anteaya> then that is the difference
15:36:49 <jgriffith> patrickeast: just from my perspective... I *thought* the point was to make it "easier" for folks to deploy/run CI and have commonality so those that need help can get it
15:37:01 <patrickeast> jgriffith: exactly!
15:37:07 <anteaya> folks whose level of comprehension includes reading logs can handle the information in the context it is offered
15:37:10 <patrickeast> jgriffith: and what i'm advocating for is to make it easier
15:37:25 <jgriffith> patrickeast: yes, but NOT to say... this is the only way to do it and we shouldn't expect things to be able to be deployed real-time
15:37:29 <anteaya> are we going to spend all meeting on this?
15:37:38 <jgriffith> anteaya: hehe... sorry :)
15:37:43 <anteaya> just checking in case anyone has anything else they want to talk about
15:37:43 <patrickeast> anteaya: sorry, didn't mean to hijack the meeting
15:37:52 <anteaya> jgriffith: not at all, glad you are here
15:37:53 * jgriffith is being quiet on that topic now
15:38:02 <anteaya> jgriffith: no no no, stay
15:38:05 <BobBall> What I'd like to see is the 3rd party CI operators coming up with a set of scripts + (updating) set of tags for a known-good CI.  That doesn't need to be an Infra recommendation, as anteaya points out
15:38:13 <anteaya> it is just that we just launced into this
15:38:13 <jgriffith> anteaya: Oh, I'll stay for sure, just letting that topic go :)
15:38:22 <anteaya> without me getting a clear sense of the room
15:38:38 <anteaya> so I still don't know if people had other things or want to use all the time for this
15:38:59 <anteaya> BobBall: agreed and good summary
15:39:18 <anteaya> operators talking to operators is a great dynamic
15:39:27 <asselin> from my point of view, first get everything on the same scripts on master, then add e.g. toggle switch to use latest released/tagged version.
15:39:39 <anteaya> but my default is looking to where we point people for support
15:39:55 <BobBall> agreed asselin
15:40:02 <anteaya> so happy to point them to other operators for those who don't understand
15:40:10 <anteaya> asselin: yup
15:40:18 <anteaya> asselin: that sounds good to me too
15:40:35 <anteaya> posts to the mailing list usually just send folks to infra channel
15:40:55 <anteaya> so having a play to address that flood when it happens is good to have in place
15:41:47 <anteaya> so again, very supportive of the early direction of the conversation, which was y'all figuring out what versions work best for you
15:42:37 <asselin> ok I started this topic....I'm done
15:42:47 <anteaya> thanks asselin, a good conversation
15:42:55 <anteaya> does anyone have anything more on this topic?
15:43:30 <anteaya> does anyone have anything else they would like to discuss?
15:44:23 <anteaya> patrickeast: so you were going to submit a patch to puppet-openstackci to include a version parameter?
15:44:51 <patrickeast> anteaya: yes
15:44:57 <anteaya> great thank you
15:45:11 <anteaya> does anyone have any reason why I shouldn't close the meeting today?
15:45:48 <anteaya> thank you everyone for your kind attendance and participation
15:45:52 <anteaya> see you all next week
15:45:56 <anteaya> #endmeeting