15:00:32 <mattmceuen> #startmeeting openstack-helm
15:00:33 <openstack> Meeting started Tue Nov 28 15:00:32 2017 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:37 <openstack> The meeting name has been set to 'openstack_helm'
15:00:37 <jayahn> o/
15:00:38 <mattmceuen> #topic rollcall
15:00:38 <srwilkers> o/
15:00:44 <mattmceuen> y'all are fast this morning!
15:00:57 <mattmceuen> I've never seen a group of folks so eager for a meeting ;-)
15:01:12 * srwilkers blinks
15:01:20 <korzen> hello
15:01:30 <mattmceuen> hi korzen!
15:01:53 <mattmceuen> Agenda is here:  https://etherpad.openstack.org/p/openstack-helm-meeting-2017-11-28
15:02:05 <mattmceuen> We'll give folks a couple minutes to filter in and add anything to the agenda
15:02:28 <mattmceuen> We'll be doing something slightly different today
15:03:12 <mattmceuen> This will be the first of a monthly meeting focused on CICD and gating-related topics, since those are so critical to QA and require quite a bit of coordination across folks
15:03:21 <SamYaple> o/
15:03:32 <mattmceuen> And also to understand different CICD use cases, and help share ideas and best practices
15:03:59 * jayahn yeah~
15:05:10 <mattmceuen> #topic Round table, overview of CI in OSH
15:05:22 <MarkBaker> o/ first timer - be gentle....
15:05:22 <lamt> o/
15:05:31 <mattmceuen> Welcome MarkBaker :)
15:05:45 <portdirect> MarkBaker: o/
15:06:10 <mattmceuen> portdirect, want to kick this off with a status / overview of the new zuul3 gate work?
15:06:55 <robertchoi80> hi
15:07:08 <portdirect> sorry, got distracted
15:07:23 <portdirect> s the zuulv3 work has been taking two prongs
15:07:33 <portdirect> 1) splitting of the charts into diff repos
15:07:45 <portdirect> 2) converting to ansible (and voting!) gates
15:08:11 <portdirect> this has allowed us to treat-osh infra a bit more roughly than we would have the main repo
15:08:21 <portdirect> and I hope its in quite a good place now
15:08:36 <portdirect> we have some people working on the docs for it now
15:08:52 <portdirect> but it should be a lot more robust than the old kubeadm-aio hack
15:09:16 <portdirect> we have used this as the basis for setting up k8s for osh
15:09:31 <portdirect> and then moved the legacy scripts to run under this cluster
15:10:07 <portdirect> with the next step/stage being importing the 'chart-runner' from osh-infra to allow us to launch all services via ainsible
15:10:20 <portdirect> and check upgrades function as expected
15:10:41 <portdirect> once this is done - I think the biggest item remaining will be to test cluster restarts in the gate
15:10:55 <portdirect> as currently this is the biggest weakness of osh
15:11:05 <mattmceuen> upgrade gates are going to be a great thing.
15:11:16 <jayahn> agreed
15:11:43 <MarkBaker> portdirect, docs referred to are at https://docs.openstack.org/openstack-helm/latest/ ?
15:11:44 <portdirect> jayahn: have you peeps been testing node/cluster restarts in your pipelines?
15:11:54 <jayahn> "test cluster restart" means restarting everything?
15:11:55 <portdirect> MarkBaker: these are the current docs
15:11:59 <mattmceuen> so to summarize a little of that -- the openstack-helm-infra gates set up k8s platform, and today the openstack-helm gates call the legacy scripts to deploy the osh charts
15:12:23 <portdirect> MarkBaker: this is the *very early* wip on the new docs: https://review.openstack.org/#/c/523173/
15:12:32 <mattmceuen> but in the near future the openstack-helm gates will be continue to be enhanced
15:12:42 <MarkBaker> portdirect, thx
15:12:55 <jayahn> portdirect: not in pipeline, but definitly testing it manually to see what problems are there, how to solve it.
15:14:34 <mattmceuen> before we move on to some other CI topics -- any questions on the existing state of CI in the OSH projects themselves?
15:14:57 <jayahn> just need to learn new CI.
15:14:59 <jayahn> :)
15:15:36 <mattmceuen> "make dev-deploy" is a neat new feature to learn in osh-infra ;-)
15:15:58 <mattmceuen> #topic container images
15:16:01 <portdirect> jayahn: i'd really appreate you guys having a look through these: https://github.com/openstack/openstack-helm-infra/tree/master/tools/gate/playbooks/deploy-helm-packages/tasks , https://github.com/openstack/openstack-helm-infra/blob/master/tools/gate/chart-deploys/default.yaml
15:16:39 <portdirect> as they are what I'd like to bring into osh itself for launching/managing charts
15:16:43 <portdirect> SamYaple: you about?
15:16:54 <jayahn> robertchoi80: need to check things portdirect throw us. :)
15:16:59 <SamYaple> portdirect: yes
15:17:05 <portdirect> So on images, we now have a new option :) LOCI
15:17:09 <robertchoi80> okay
15:17:11 <StaceyF> Kaspars has started the Jenkinsfile for loci builds - https://review.gerrithub.io/#/c/388686/
15:17:14 <mattmceuen> For some of the newer folks to the team -- one of the design goals of OSH is to be able to support multiple container image sets
15:17:35 <mattmceuen> E.g. Kolla and now LOCI.  OSH aims to be largely agnostic.
15:17:40 <portdirect> I am running these internally, and our CI/CD team lead by StaceyF is building out infra for internal builds
15:18:01 <jayahn> okay. gerrithub.
15:18:02 <portdirect> It would be great if we could start cross gating with LOCI
15:18:12 <portdirect> SamYaple: how does that sound? :D
15:18:22 <SamYaple> sounds good for sure
15:18:26 <jayahn> portdirect: StaceyF: does gerrithub needs any permission to see logs from ci?
15:18:33 <StaceyF> Just a github account
15:18:38 <SamYaple> with zuulv3 LOCI can just reconsume OSH jobs
15:19:22 <jayahn> when it fails, it throw us http://10.24.20.18:30085...... which i cannot see
15:19:23 <portdirect> Once we have that in place, I'd like to consider changing the default image to LOCI, and provide over-rides/gates for kolla as well
15:20:01 <StaceyF> The Jenkinsfile will be something that you would use in your internal Jenkins and you will just need to update variables to publish them wherever needed.
15:21:01 <robertchoi80> staceyF: that sounds good.
15:21:26 <StaceyF> We will be allowing access to our LOCI images via Artifactory
15:21:39 <StaceyF> Should happen this week
15:22:21 <StaceyF> Jayahn we'll update the 30085 to our nexus server
15:24:28 <mattmceuen> Team - I'm going to go out of order on the Agenda -- we have a lot to get through and there a few things I want to make sure we get to today
15:25:07 <mattmceuen> #topic Cookiemonster from skt
15:25:18 <mattmceuen> (but hopefully we'll be able to get to everything today)
15:25:28 <jayahn> we got promoted. :)
15:25:28 <mattmceuen> jayahn, want to give us an overview of cookiemonster?
15:25:32 <mattmceuen> :-D
15:25:36 <jayahn> robertchoi80 will do
15:26:06 <jayahn> robertchoi80: is our ci guy
15:26:16 <robertchoi80> all right. I'll briefly explain that
15:26:28 <mattmceuen> robertchoi80 is StaceyF's new best friend
15:26:37 <jayahn> yes!
15:26:42 <StaceyF> yes we met in Sydney :)
15:28:03 <robertchoi80> it's ha test tool that terminates varios k8s resources (such as deployment or daemonset) on regular interval and duration.
15:28:18 <robertchoi80> it's similar to chaosmonkey from netflix.
15:28:29 <robertchoi80> for eg,
15:29:45 <robertchoi80> it tries to pick randomly one pod from the deployed resources, and if the number of replica is more than 1, it kills one of them.
15:30:03 <mattmceuen> can you give us a link to cookiemonster -- it's not the first hit on google :)
15:30:20 <robertchoi80> https://github.com/sktelecom-oslab/cookiemonster
15:30:47 <jayahn> we just uploaded our source code, documentation and more example will come
15:30:57 <portdirect> robertchoi80: what are the primary differences between it and https://github.com/asobti/kube-monkey, https://github.com/linki/chaoskube ?
15:30:59 <mattmceuen> awesome
15:31:37 <StaceyF> very cool
15:32:39 <robertchoi80> portdirect: sorry, I am not the one who wrote that tool, so I can't answer that right now. I'll give you the answer later by email
15:32:52 <robertchoi80> that tool = cookiemonster
15:33:05 <jayahn> portdirect: we probably need to do a quick comparison to answer your question. several month ago when we decide to write our own code, these tools were not perfect, so we decided to quickly make our own.
15:33:42 <mattmceuen> def interested in that comparison.  We'd been wanting to use kubemonkey or chaoskube, but haven't gotten too far with that yet
15:33:44 <jayahn> and had master plan to break anything via cookiemonster. :)
15:34:20 <jayahn> okay. we will update with some comparison.
15:34:40 <portdirect> jayahn: sounds good - I think i remember you mentioning being able to target specific applications at the PTG but a simple table would be a really great addition to the readme
15:34:53 <jayahn> we don't have a cookiemonster chart yet, but will definitly make one soon and put on openstack-helm-addons for anyone to use
15:35:03 <mattmceuen> Thanks robertchoi80 & jayahn -- looks very promising from the readme, and would love to learn more
15:35:08 <portdirect> jayahn: that would be awesome!
15:35:23 * portdirect is lazy and likes cookies
15:35:39 <mattmceuen> moving on:
15:35:43 <mattmceuen> #topic Where do we want to go next, how much can we share?
15:35:59 <mattmceuen> "where do we go next" -- always a good question.
15:36:01 * jayahn i got a biggest difference. cookiemonster is way cooler name than others
15:36:19 * mattmceuen agrees 100%
15:36:39 <srwilkers> jayahn: +++++
15:36:49 <srwilkers> however, kube-monkey wont eat my cat
15:37:16 <mattmceuen> First, from an AT&T perspective, like StaceyF showed with our CI - we want to share everything but the configuration, or anything that's irrelevant outside of AT&T
15:37:48 <StaceyF> We will share our Jenkins configurations of any plugins, just no passwords :)
15:38:07 <mattmceuen> so everyone please feel free to either contribute, or take whatever looks applicable and adapt for your use!
15:38:17 <portdirect> So - I covered where i'd like to see our community gates go a bit above, but I think we also need to work out a set of critera for plugins/alternative backends
15:38:39 <portdirect> eg OpenContrail and friends
15:38:49 <portdirect> anyone from Juniper here?
15:39:18 <Krishna> yes, this is Krishna from Juniper
15:39:29 <mattmceuen> Welcome Krishna!
15:39:37 <portdirect> Krishna: o/
15:39:40 <Krishna> Hello Matt
15:40:27 <portdirect> have you peeps worked out what infra will be needed to test opencontrail? can we do it with simplegateway, to provide parity with the current gate tests?
15:41:04 <portdirect> and would this be able to run in the community provided infra? or would a third party gate be required?
15:41:19 <portdirect> i think youve had a few people (andrey?) looking into this
15:42:06 <Krishna> We have done work with the simple GW to provide some parity and should be available to run in the community infra as stated.
15:42:57 <Krishna> Andrey Pavlov has done some tests and there are some pending items which we need to hash out with Andrey.
15:43:15 <mattmceuen> that is great to hear.  do you have that work somewhere for us to have a look at?  Is it ready for that?
15:43:39 <portdirect> Krishna: sounds good - could we make documentation a priority, which will help us help you guys get there
15:44:33 <jayahn> from skt perspective, we can share jenkins configuraiton, jenkinsfiles, along with some whitepaper on how we do it.
15:44:33 <jayahn> However, I personally think we (skt) need to learn osh-gate more seriously.
15:44:48 <Krishna> Matt: yes, we will send an emails with the compeleted work
15:44:58 <mattmceuen> Awesome, thanks Krishna.
15:45:07 <portdirect> Krishna: a PS is always good too ;)
15:45:09 <Krishna> PortDirect: we will make the documentation a priority and post it soon.
15:45:32 <jayahn> to figure out (in long-run), how close/similar we would like to do with osh community gate.
15:45:36 <mattmceuen> Our goal w.r.t. OpenContrail gating is 1) to make it a first class citizen of OSH QA gates
15:45:55 <Krishna> portdirect: regarding the engaging the PS, we are in discussions for that.
15:46:19 <mattmceuen> 2) to move toward OpenContrail installation on OSH being a first class installation mechanism for k8s that is well-documented and simple to use
15:48:11 <mattmceuen> Thanks, Krishna.  I'm also planning to come to the OpenContrail meeting @ KubeCon, and would like to discuss OSH there too
15:48:56 <mattmceuen> jayahn -- yeah, please let us know your feedback on the new OSH gating
15:49:43 <mattmceuen> jayahn -- do you have gating for sona yet?  Is that open, nd something we could learn from as well?  May be good reusable principles
15:50:13 <robertchoi80> matt: that's still in our to-do list.
15:50:24 <mattmceuen> ok cool
15:50:28 <robertchoi80> should be done in next 2 weeks
15:50:29 <jayahn> just did not have time. :(
15:50:44 <jayahn> we have a release cut this week. super busy. :)
15:50:46 <mattmceuen> nice!  2 weeks ain't bad
15:51:36 <jayahn> fyi, it will be provided as 3rd party gating.
15:51:45 <mattmceuen> fanstastic
15:51:57 <mattmceuen> alrighty -- catching up on topics I skipped :D
15:52:02 <mattmceuen> #topic Heat Templates
15:52:12 <mattmceuen> StaceyF take it away
15:52:28 <portdirect> jayahn: thats great - I'd really like to have SONA up there as a 1st class citizen as well - bizarrely it (via CORD) was one of the projects that got me into this....
15:53:31 <StaceyF> I had already mentioned it earlier, we'll be putting up a patchset with all of the heat templates that are used in our Jenkinsfile.
15:53:57 <mattmceuen> awesome, thanks, I missed that in the flurry
15:54:30 <mattmceuen> superseed though - I don't think I missed that
15:54:39 <mattmceuen> #topic SuperSeed
15:55:38 <StaceyF> The superseed job is triggered via a change to a Jenkinsfile or seed.groovy file.  It updates any pipeline that has a change.  We use this in our Development environment for Jenkins to verify that the seed and Jenkinsfile work before putting it into our Prod Jenkins
15:55:59 <StaceyF> https://github.com/att-comdev/cicd/tree/master/cicd/SuperSeed
15:56:20 <StaceyF> This is the job that will be updated to allow logs accessible by everyone
15:58:26 <mattmceuen> Thanks StaceyF.  Any q's on this topic?
15:58:36 <StaceyF> The goal will be to have this one job be in a Jenkins chart that you can kick it off and it builds all of your pipelines for osh
15:59:18 <mattmceuen> We're about out of time, all -- clearly this focused meeting was a good idea.  Do you all want to 1) save the remaining few topics for next meeting, or 2) till the next CI meeting in a month?
15:59:25 <mattmceuen> I'm leaning toward next meeting
15:59:45 <jayahn> can use some of next meeting time.
15:59:57 <mattmceuen> cool beans - I'll copy them over
16:00:15 <mattmceuen> Thanks, everyone!  see you in the OSH chat!
16:00:18 <mattmceuen> #endmeeting