16:00:49 <VW_> #startmeeting Large Deployment Team
16:00:50 <openstack> Meeting started Thu Dec 18 16:00:49 2014 UTC and is due to finish in 60 minutes.  The chair is VW_. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:53 <openstack> The meeting name has been set to 'large_deployment_team'
16:01:12 <jlk> o/
16:01:22 <belmoreira> hey
16:01:40 <VW_> hello folks.  Thanks for joining
16:01:51 <VW_> if anyone needs it, I set the following up for notes, etc
16:01:53 <VW_> https://etherpad.openstack.org/p/Large-Deployment-Team-Meetings
16:02:51 <VW_> Let's dive in to the first thing on there
16:03:01 <VW_> #topic Midcycle Meetups
16:03:41 <VW_> I've seen a little chatter about some of the midcycles.  In Paris, we discussed getting representatives from this group to as many as we could
16:03:50 <belmoreira> who is going?
16:03:52 <VW_> anyone already planning on getting to one or more of them
16:04:17 <jlk> I'm trying to convince my current management that I need to go to the nova mid cycle
16:04:41 <belmoreira> VW_: I'm not planning to go
16:05:26 <VW_> I know we have several devs going to Nova.  I may have to lean on them for insight, feedback, etc for budget reasons as well
16:05:39 <VW_> for now, I've got jlk down as a maybe in the notes
16:06:51 <VW_> are there any others we care strongly about
16:07:00 <VW_> I'm assuming Neutron mid cycle
16:07:04 <VW_> what else?
16:07:12 <jlk> good question.
16:07:29 <jlk> Is glance having one?
16:07:38 <jlk> I know as deployers/operators we have... issues with glance
16:07:48 <sigmavirus24> jlk: glance is
16:07:55 <sigmavirus24> I can't remember exactly where though in all candor
16:07:56 <jlk> and it would be nice to get an idea on how things are progressing to get glance-api out of the data transfer game.
16:08:27 <belmoreira> I don't think ceilometer has one... for us is one of the most problematic components
16:08:51 <sigmavirus24> jlk: https://etherpad.openstack.org/p/kilo-glance-mid-cycle-meetup
16:09:34 <VW_> hmm belmoreira - good point - we'll need to ask around about that
16:11:48 <VW_> Ok, so from what we've discussed here, it sounds like we'd love reps and/or insight to Nova, Glance Neutron and Ceilometer (if one materailizes)
16:12:11 <VW_> right now jlk is our only active Large Deployment Team member that might be going to one
16:12:59 <VW_> I'll leverage some of our more Ops minded core devs at the nova one as well and I may know some folks in the loop on the glance mid cycle
16:13:07 <VW_> let me assign myself a follow-up action
16:13:37 <VW_> #action VW_ follow up with RS folks that might know about/be attending Glance mid cycle
16:14:22 <VW_> Also, I bugged Tom about the Ops mid cycle a few days ago.  Sounds like details are still pending
16:14:37 <jlk> yeah that's another we should try to attend.
16:15:40 <VW_> Ok - well it sounds like there is plenty of follow-up with regard to mid cycles.  Anything else before we move on?
16:16:37 <VW_> I'll go with no :)
16:16:48 <VW_> #topic Periodic Tasks
16:17:22 <VW_> So one of the things we discussed as a "quick win" in Paris was providing collective feedback on periodic tasks we regularly tweak/disable
16:17:32 <VW_> did anyone bring some examples with them?
16:17:53 <jlk> I did not, we're not currently tweaking them at all.
16:18:19 <belmoreira> well we are disabling: heal_instance_info_cache_interval
16:18:23 <mdorman> only one i can think of off the top of my head is the intance info cache update interval
16:19:27 <belmoreira> the main reason for us is our infrastructure design
16:19:58 <belmoreira> it can be really heavy in the nova-network nodes, and we chose to have few of them...
16:20:06 <VW_> I think we adjust that one too, belmoreira, but I'm admittedly behind in my homework
16:20:20 <VW_> same for mdorman's
16:20:37 <VW_> rabbit queues can get messy with a lot of the every minute updates
16:21:24 <VW_> mdorman: can you elaborate on what you change and why with respect to that one?
16:22:03 <mdorman> i think we change it to 15 minutes or something.  and tbh we only adjusted it because we heard others had issues with it, and we wanted to try to address it before it became a problem
16:22:18 <VW_> fair enough
16:22:23 <mdorman> but yeah, intuitively, 60s updates on 100x VMs, is not such a great thing.
16:22:37 <mdorman> s/100x/100s/
16:23:31 <VW_> tangent - I know belmoreira uses cells.  Do you mdorman?
16:24:00 <jlk> yeah, that's interesting data. Cells vs no, and conductor vs no.
16:24:03 <VW_> we've actually had to go to multiple global cells workers and a lot of the driver is the overall volume of updates from the computes
16:25:07 <klindgren> We are in the process of switching to cells.  But currently we are not cells
16:25:08 <mdorman> we are working on converting to cells right now.  but don’t have any prod/larger scale experience with them yet.
16:26:20 <VW_> good to know klindgren / mdorman - happy to share insight as you move along.  While not directly related to periodic tasks, I bring it up because of the reasons mentioned above
16:26:55 <VW_> jlk - does condcutor make this whole line of conversation moot since a "smaller" number of hosts can send their task update info to a collection point?
16:26:56 <klindgren> We do use conductor though
16:27:38 <belmoreira> For those looking into cells there is a working group for cellsv2. See: https://wiki.openstack.org/wiki/Meetings/NovaCellsv2#Agenda
16:28:23 <VW_> really, klindgren - that's great - anything you can share relative to conductor and periodic tasks
16:28:39 <VW_> thanks belmoreira - I keep trying to make that one and my schedule is not co-oprating
16:28:48 <jlk> VW_: I honestly don't know.
16:29:01 <jlk> VW_: conductor makes things nicer when it's computes directly writing to databases
16:29:08 <jlk> but I don't think the periodic tasks are that.
16:29:09 <VW_> going to put down and agenda item for next meeting to dive into current status of cells V2 for this group, belmoreira if you think it's worth it
16:29:20 <VW_> good point, jlk
16:29:52 <klindgren> Not really - honestly we aren't at a point, yet, where we are big enough to run into some of these scaling issues.
16:29:53 <mdorman> yeah i would love more detials on cells v2, i too haven’t had the time/schedule to really get up to speed on it
16:30:36 <belmoreira> VW_: definitely. I think this group should be informed and participate in the discussions
16:30:44 <VW_> +1
16:31:14 <VW_> Ok, so it sounds like we have a little feedback on periodic tasks.  Perhaps take a summary of this chat to the mailing list?
16:31:23 <jlk> +1
16:31:38 <belmoreira> +1
16:31:48 <jlk> is there a separate large deployers list or are we using the existing operators list? (prefer the latter)
16:32:04 <VW_> the decision in Paris was use ops list but tag emails
16:32:12 <jlk> nod
16:32:16 <belmoreira> I also the: bandwidth_poll_interval
16:32:16 <VW_> I've been using [Large Deployment Team]
16:32:25 <belmoreira> at the moment the default is 600
16:32:33 <VW_> ah yes -that one - how could I forget, belmoreira
16:32:55 <belmoreira> but maybe it should be disabled... I don't see many people using it...
16:33:11 <VW_> I tend to agree belmoreira
16:33:14 <jlk> what'd be nice is a break down of A) what people alter, and B) what the impact of changing the default is
16:33:29 <VW_> will work that into the email conversation
16:33:30 <jlk> IE what purpose does that periodic task serve, and how is that purpose altered by adjusting the interval.
16:35:31 <VW_> cool - we'll take that to the list then.  maybe even back it with an ehterpad for better notes
16:35:50 <VW_> moving along
16:35:59 <VW_> #topic Meeting Times
16:36:30 <belmoreira> In the docs: "Interval to pull network bandwidth usage info". However I don't know how useful it is, besides the queries that introduces...
16:37:03 <jlk> right, that's kinda the problem. The definition is somewhat circular and doesn't describe what that data is USED for.
16:37:18 <VW_> I'd like to lock down how we manage this meeting on an on-going basis.  We agreed in Paris that once a month seemed right.  With only one under our belt, I'm not ready to change that, but figured we can start there
16:37:23 <VW_> noted belmoreira, jlk
16:37:30 <belmoreira> jlk: yes
16:37:55 <jlk> VW_: once a month seems about right. The more regular operators meeting can be used for whatever else comes up.
16:38:02 <mdorman> +1
16:38:09 <VW_> agreed
16:38:30 <VW_> speaking of which - did any of you make that yesterday?  I had stuff at home I had to deal with
16:38:40 <klindgren> I did
16:39:05 <VW_> Not to get too off-topic, but anything substantial worth revisiting here, klindgren
16:39:23 <klindgren> mainly around some packaging - then some other stuff that I am totally drawing a blank on
16:39:35 <klindgren> :-)
16:39:51 <VW_> no worries
16:39:53 <mdorman> and the sample config file topic, which is loosely related to packaging
16:40:06 <mdorman> also we were trying to figure out if/when there’s a mid-cycle ops meetup
16:40:21 <mdorman> but i think mfisch/fifieldt are working on that one
16:40:28 <VW_> yeah, that seems to be the burning question.  Based on what I've heard, it's yes
16:40:33 <VW_> just when and where is still in flight
16:40:42 <VW_> but I'll try to catch Tom this evening
16:40:51 <mdorman> that’s my understanding too.  there was an action to post curent status on all that to the ML
16:41:06 <VW_> cool
16:41:58 <VW_> back to our meeting is 3rd Thursday as good as anything for you all?
16:42:07 <mdorman> works for me
16:42:13 <VW_> I picked this one to chat prior to holidays, but I don't see why we don't stick to it
16:42:23 <jlk> works well for me, that's typically my work from home day
16:43:00 <VW_> awesome
16:43:17 <VW_> I'd like to flip the time every other month though so we can catch more timezones
16:43:27 <VW_> anyone have serious concerns with that?
16:43:32 <mdorman> nope
16:43:55 <jlk> no
16:43:56 <belmoreira> sorry, I'm lost... 3rd Thursday?
16:44:23 <belmoreira> I understand now... :)
16:44:23 <mdorman> 3rd thursday of the month
16:44:30 <VW_> yes, sorry for not being clear, belmoreira
16:44:39 <emagana> sorry, quite late!
16:44:57 <VW_> no problem, emagana
16:45:03 <VW_> welcome
16:45:26 <emagana> VW_: Representing Workday! We have a huge deployment going on!
16:45:56 <jlk> VW_: oh there was one thing from the operators meeting yesterday
16:46:18 <VW_> awesome - glad you could join us.  we were just finalizing the structure for on-going meetings, emagana
16:46:24 <VW_> go for it jlk
16:46:39 <jlk> VW_: a call for deployers to write up SuperUser stories to describe how we each deal with code, how we get it from upstream, how we add any downstream changes, and how we turn it into something we put out into (pre)production
16:46:51 <jlk> A longer format show and tell
16:47:22 <jlk> While we don't think we can reach convergence on ways to do this, the community would benefit from having more explicit examples put out there, particularly when some are done using open source software
16:48:18 <belmoreira> jlk: completely agree.
16:48:19 <VW_> very nice - and noted, jlk - we also agreed in Paris that this group should get active in SuperUser.
16:48:59 <VW_> and I agree with your assessment, jlk - but you probably knew that ;)
16:49:17 <jlk> I just assume everybody always agrees with me
16:49:38 <klindgren> lol
16:49:45 <VW_> so this time slot was pretty popular.  We'll keep it as the Eurpoean friendly meeting time every other month starting in Feb.
16:50:10 <VW_> and I'll send out a doodle to pick a time in the Evening US that brings in more of APAC
16:50:52 <VW_> try to make so that if belmoreira is up smashing atoms late into the night he might be able to join as well :)
16:51:20 <belmoreira> VW_: yeah :)
16:51:57 <VW_> ok - cool.  will get that out asap then so I can get stuff locked down in the wiki's etc
16:52:31 <VW_> and not have to rely as much on email - which emagana may have pointed out I goofed up in etherpad notes
16:53:23 <VW_> we are running low on time - which brings up my last question for this meeting schedule on-gong
16:53:32 <VW_> s/on-gong/on-going
16:53:35 <VW_> is an hour enough?
16:54:04 <mdorman> i think so. and if not, we should cap it
16:54:25 <jlk> agreed
16:54:26 <mdorman> we all have enough meetings as it is
16:54:31 <belmoreira> mdorman: +1
16:54:38 <mdorman> and these can be pretty efficient
16:55:01 <VW_> sounds good.  In theory, I like us having a bit of pressure to take unresolved/longer discussions to the list anyway - especially since we will have split meeting times with varying attendance
16:55:17 <VW_> ok - cool - it stays at an hour then
16:55:48 <VW_> ok - I want to honor at least one other agenda item before we run out of time
16:55:55 <VW_> #topic Blueprints
16:56:08 <VW_> anyone have any they would like to bring to the attention of the group?
16:56:32 <jlk> I did not do good homework here.
16:57:00 <VW_> any important ones with cells v2 we should read over, belmoreira?
16:57:54 <belmoreira> There are some... let's see...
16:58:30 <belmoreira> https://review.openstack.org/#/c/136490/1
16:58:46 <VW_> #action VW_ send doodle out for alternative meeting time to support APAC members
16:59:00 <belmoreira> this is for cells migration
16:59:04 <VW_> #action VW_ get meeting details into wiki, etc now that it's been decided
16:59:41 <VW_> got it - thanks, belmoreira
17:00:06 <belmoreira> this etherpad about scheduling requirements is also important: https://etherpad.openstack.org/p/nova-cells-scheduling-requirements
17:00:24 <VW_> looks like we are at time.  if you have any more, belmoreira add them here - https://etherpad.openstack.org/p/Large-Deployment-Team-Meetings and we'll draw attention on the mailing list
17:00:50 <VW_> thank you all for joining today!  It was a little bumpy, but I'm happy with our first attempt
17:00:58 <VW_> feel free to bounce me any feedback you ahve
17:01:01 <mdorman> thanks VW_
17:01:14 <klindgren> thanks
17:01:15 <belmoreira> thanks
17:01:24 <klindgren> I do have a question for people using ceilometer at scale
17:01:35 <klindgren> have you noticed that it hammers the hell out of the nova api?
17:01:41 <VW_> I'll grab all the notes and get a summary out
17:01:51 <VW_> #endmeeting