16:02:06 <VW> #startmeeting Large Deployments Team
16:02:07 <openstack> Meeting started Thu Feb 16 16:02:06 2017 UTC and is due to finish in 60 minutes.  The chair is VW. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:11 <openstack> The meeting name has been set to 'large_deployments_team'
16:02:36 <VW> who all do we have this month?
16:03:00 <mdorman> o/
16:03:01 <klindgren> o/
16:03:18 <jamemcc> o/
16:03:46 <VW> excellent.
16:03:59 <admin0> o/ - new addition
16:04:07 <VW> welcome!
16:04:23 <VW> and jamemcc - glad you could make it
16:04:30 <VW> along with the usual folks
16:04:40 <jamemcc> FYI - I'm the chair of the new LCOO workign group and so I'd like to collaborate on items in common.
16:04:46 <VW> yep
16:04:55 <VW> that will tie in to a topic in a bit ;)
16:05:03 <jamemcc> But hopefully I'll also be able to contribute just as me as well.
16:05:17 <VW> #topic Ops Midcycle
16:05:31 <klindgren> I will not be there
16:05:56 <VW> me either, unfortunately
16:06:21 <VW> i was hoping belmoreira might be in the meeting today to see if he was planning on attending
16:06:27 <admin0> odd timing though .. it starts when office closes
16:07:11 <VW> not sure I follow admin0
16:07:42 <admin0> sorry .. wrong window
16:07:54 <admin0> lets continue
16:07:55 <VW> ah - make sesnse :)
16:09:19 <admin0> so this meeting, large deployment team .. is it about intenal openstack deployment ( like for the foundation) or about people with large deployments
16:09:26 <admin0> and what is considred large
16:09:29 <VW> It doesn't sound like anyone here is going to the mid-cycle.  I'll try to follow up with some other team members. I just wanted to make sure there wasn't anything specific the teamm wnated input/attendance in
16:09:44 <VW> admin0: it's for people with "large" deployments
16:09:54 <VW> but that definition is a bit open
16:09:56 <mdorman> for whatever “large” means to you.
16:10:00 <belmoreira> VM sorry I'm late
16:10:09 <VW> hey hey belmoreira - welcome!
16:10:14 <mdorman> basically thinking about scale issues, etc.
16:10:19 <VW> for some, admin0 - large is a big deployment
16:10:27 <VW> for others, it's many many small deployments
16:10:38 <admin0> i am approaching 100 compute nodes per availabiltiy zone,   so 200 in total ; approx 6000+ instances
16:10:40 <VW> we have acive membes with both scnarios
16:10:41 <admin0> do i count in ?
16:10:58 <admin0> and i put down my issues :D ?
16:11:10 <admin0> \o/ — outgoing DDoS, syn flood etc
16:11:16 <admin0> how to handle those
16:12:06 <klindgren> We joined LDT when we had aboutthat many compute nodes
16:12:25 <admin0> so what kind of contribution do i provide ?
16:13:07 <VW> yeah admin0 - sometimes is as much about finding common issues as you move through various phase shifts - 10's to 100's, 100's to 1000's, 1000's to 10,000's etc
16:13:18 <VW> as it is to the size you are today
16:14:08 <admin0> my first time in the meeting, so i will read others interacting and then will know how to contribute ..
16:14:15 <admin0> please continue
16:14:23 <VW> actually admin0 - historically, the team has partnered with dev teams to get blueprints and work going on scale issues we have in common
16:14:36 <VW> but yes, I think you'll find ways to jump in
16:14:39 <admin0> ok
16:14:41 <VW> belmoreira: are  you going to Milan
16:15:47 <belmoreira> unfortunately not... but two CERN folks will be there
16:16:33 <belmoreira> they will look more for ironic experiences
16:17:19 <VW> OK
16:17:45 <VW> I will have some fellow Rackers there too from the private cloud side, but it looks like none of LDT is going
16:18:15 <VW> was going to make sure there wasn't anything specific that we needed looked in to/attended as a team
16:19:27 <VW> on to the next thing then
16:19:32 <VW> #Boston
16:19:39 <VW> oops
16:19:43 <VW> #topic Boston
16:20:29 <VW> With only a couple of months left, I wanted to circle back on forum sessions
16:21:02 <VW> and I apologize, I haven't pulled up the notes from our sessions in BCN yet due to a busy morning
16:21:09 <VW> Give me a minute
16:25:32 <VW> ok - sorry - found it
16:25:34 <VW> #link https://etherpad.openstack.org/p/BCN-Large-Deployments-Team
16:26:36 <VW> I have requested some specific meeting times for the team, but I also wanted us to circle back and think about forum session topics we may want to partner with others on
16:27:27 <klindgren> So cellv2 and conversion from cellsv1 - would be interesting
16:27:49 <VW> good call.
16:28:10 <klindgren> ALso neutron scaling.  We just recently had issues with neutron and dhcp agent's
16:28:20 <klindgren> bascially blowing up neutron with large provider networks
16:28:40 <VW> that's good too
16:28:59 <klindgren> due to dhcpagents getting shifted around and neutron reprograming all of the security group flows on the vm's attached to those networks
16:29:01 <VW> belmoreira: is this the time to try and get conversation going on some of the Glance issues we've been chasing
16:29:15 <klindgren> since we run with /22's its like 1k reconfigs at a pop
16:29:53 <klindgren> Which pounded the db, the api servers and then took out all of the nuetron agents
16:30:01 <belmoreira> I think so... and what is the role of glare
16:30:07 <klindgren> due to the api's not processing requests fast enough and agent down time kicking in
16:30:22 <klindgren> which then caused a whole cloud security group refresh
16:30:25 <klindgren> it was....
16:30:27 <klindgren> great fun
16:30:54 <belmoreira> neutron scalability would be interesting
16:31:04 <belmoreira> I also would be interested in cellv2 experiences in newton/ocata and the placement api
16:31:33 <klindgren> Also - I havent stayed abreast of searchlight
16:31:44 <klindgren> and nova's desire to use that for listing of vm's across cells.
16:31:55 <VW> created this
16:31:58 <VW> #link https://etherpad.openstack.org/p/LDT-BOS-fourm_topics_sractchpad
16:32:04 <VW> let's put these ideas in there
16:32:26 <VW> jamemcc: any of these issues overlap with anything LCCO is discussing
16:32:39 <VW> would be nice if we could partner on a forum session sumbission or two
16:32:42 <jamemcc> Yeah - part of the missing questions for LCOO form the December session was what might LCOO be bringing in and obviously if it's the same sort of thing as LDT brings or is naturally interested/expereinced in - lets work ahead of time on the same Forum sessions.  So that said - what will we be bringing - that's really still pretty open - trying to get subteams organized to bring the user stories.  The Subteams are Networking - which might
16:32:42 <jamemcc> bring User Stories to help bring L2 and L3 networking solutions together.
16:33:56 <jamemcc> Security - which should eb bringing a lot of the user stories that large operator are experienced in implementing in thier older platforms but need to be readdressed in a new Cloud Native way.
16:34:59 <jamemcc> What we call Resource Management - which is supposed to be bringing in user stoies to help guide cross project things to help coordiante across clouds (maybe what was earlier called instances)
16:36:11 <VW> hmm - don't want to side track, but I'd be carful about equating instances to clouds
16:36:19 <VW> that's how most people refer to VMs
16:36:32 <jamemcc> Thanks - glad to get the feedback
16:36:41 <klindgren> So - I do have a general question is anyone re-doing their deployment infrastructure?
16:36:56 <klindgren> Thats something that we have been focused on is moving from packages to kolla containers
16:37:02 <jamemcc> I'll keep ttyping to get it out there but please dont' wait
16:37:03 <VW> on sec klindgren
16:37:11 <VW> one sec even
16:37:27 <VW> can everyone here dump some thoughts in the ehterpad above
16:37:50 <VW> I chatted with Tom last night, I think the brainstorming process for forum sessions starts at the end of the month
16:38:16 <jamemcc> QA/Testing - User stories about availability and anything we find ourselves automating to make things better in the upcoming release as well as the types of scriptstest we run as we are deploying individual clouds
16:38:17 <VW> we'll take some of this to mailing lists, flesh them out and then see if we can find dev partners and/or other working groups to help champion any
16:38:49 <VW> cool jamemcc - I'm sure we'll find a couple to partner on
16:39:17 <VW> in the spirit of sharing best practices, I'd like to switch topics and dive in to klindgren's questions
16:39:21 <VW> unless anyone objects
16:39:59 <VW> since the team is 1 part working on chagnes to OpenStack and one part sharing the pain/experiences/best practices :)
16:40:19 <jamemcc> Otehrs are Containerized Control Plane and what we call Operations/LMA
16:40:28 <jamemcc> Done or now
16:40:34 <jamemcc> Done for now
16:40:48 <VW> cool jamemcc - through some notes in the etherpad and we'll continue working on this
16:40:55 <VW> #topic general discussion
16:41:05 <VW> klindgren: continue... :)
16:41:52 <klindgren> Bascially the tool that we had been using for packagin our source code, we decided to get away from it.  Along with wanting to "breakup" having system level packages isntalled on the same servers
16:42:07 <klindgren> I realize that a number of larger deployment went with overcloud/undercloud modle
16:42:31 <klindgren> so they dont have a bunch of baremetal nodes with system python packages installed on them
16:42:52 <klindgren> and run into upgrade issues, because updating one component to the next release
16:43:00 <klindgren> means having to do them all at the same time on that node
16:43:41 <klindgren> So we are using kolla/docker to do the containerization and building of containers
16:43:47 <klindgren> then working on a new deployment stuffs
16:43:57 <klindgren> using ansible - but not kolla-ansible
16:44:22 <klindgren> Also worked on templating our cloud configuration into a centralized yaml repo
16:44:32 <klindgren> and taking our config management data and templatizing it
16:44:41 <klindgren> to make standing up new clouds easier
16:44:57 <klindgren> and minimize varriances between our current clouds
16:45:14 <klindgren> wondering if anyone else is doing similar work?
16:45:14 <jamemcc> Speaking more as AT&T now - not as LCOO - We are working pon a new deployment model more Kolla/Kubernetes and Helm Charts to guide it.  This is just a Proof of Concept based on earlier theories and PoCs.
16:45:31 <jamemcc> It will hopefulyl get presented and discussed at the PTG nxt week.
16:45:57 <belmoreira> I need to leave. I will follow and suggest more things in the etherpad. cheers.
16:46:04 <VW> thanks, belmoreira
16:46:18 <klindgren> We thought about doing Kube - but at some point wanted to avoid having a semi complex system
16:46:33 <klindgren> to debug ontop of trying to figure out what was broken
16:46:37 <klindgren> when things go south
16:47:05 <klindgren> IE I dont need to debug a kube issue, when my production cloud is DOA.
16:47:06 <jamemcc> I think the user stories behind this are really important to pull out
16:47:21 <jamemcc> You are getting at them
16:49:25 <VW> you are ahead of us klindgren.  I'd like to start shifting to some containers, but we have several other high priority projects, so for now, I just to settle for cleaning up what we have as a deployment model
16:49:31 <VW> since my team just inherited it
16:49:53 <VW> but yes, we are using the "undercloud" model with VMs for most of our control plane nodes
16:50:05 <VW> except for Glance APIs - big beefy boxes there
16:50:20 * VW shakes is fist at Glance APIs having to be in path for image transfer
16:51:03 <klindgren> I know lets take all the snapshot data, centralize it to a set of servers, then transfer that to yet another backend
16:53:29 <VW> so are you converted to Kolla, klindgren
16:53:33 <klindgren> neg
16:54:02 <klindgren> we haven't converted anything yet.  Just making sure all the containers work and we can start everything in a standalone env
16:54:17 <klindgren> then working on "taking over" an existing environment
16:55:51 <VW> gotcha
16:56:00 <VW> what's your rough timeline?
16:58:36 <VW> well, this was a good meeting - more work to do
16:58:50 <VW> but now my IRC and other chat stuff is blowing up with a few little fires
16:58:56 <VW> so I'm going to go ahead and call it
16:59:03 <VW> don't forget about next month's meeting
16:59:41 <VW> 3/17 at 03:00 UTC
17:00:19 <VW> I'll actually be in your neck of the woods for that, mdorman
17:00:26 <VW> #endmeeting