16:00:49 #startmeeting Large Deployment Team 16:00:50 Meeting started Thu Dec 18 16:00:49 2014 UTC and is due to finish in 60 minutes. The chair is VW_. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:51 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:53 The meeting name has been set to 'large_deployment_team' 16:01:12 o/ 16:01:22 hey 16:01:40 hello folks. Thanks for joining 16:01:51 if anyone needs it, I set the following up for notes, etc 16:01:53 https://etherpad.openstack.org/p/Large-Deployment-Team-Meetings 16:02:51 Let's dive in to the first thing on there 16:03:01 #topic Midcycle Meetups 16:03:41 I've seen a little chatter about some of the midcycles. In Paris, we discussed getting representatives from this group to as many as we could 16:03:50 who is going? 16:03:52 anyone already planning on getting to one or more of them 16:04:17 I'm trying to convince my current management that I need to go to the nova mid cycle 16:04:41 VW_: I'm not planning to go 16:05:26 I know we have several devs going to Nova. I may have to lean on them for insight, feedback, etc for budget reasons as well 16:05:39 for now, I've got jlk down as a maybe in the notes 16:06:51 are there any others we care strongly about 16:07:00 I'm assuming Neutron mid cycle 16:07:04 what else? 16:07:12 good question. 16:07:29 Is glance having one? 16:07:38 I know as deployers/operators we have... issues with glance 16:07:48 jlk: glance is 16:07:55 I can't remember exactly where though in all candor 16:07:56 and it would be nice to get an idea on how things are progressing to get glance-api out of the data transfer game. 16:08:27 I don't think ceilometer has one... for us is one of the most problematic components 16:08:51 jlk: https://etherpad.openstack.org/p/kilo-glance-mid-cycle-meetup 16:09:34 hmm belmoreira - good point - we'll need to ask around about that 16:11:48 Ok, so from what we've discussed here, it sounds like we'd love reps and/or insight to Nova, Glance Neutron and Ceilometer (if one materailizes) 16:12:11 right now jlk is our only active Large Deployment Team member that might be going to one 16:12:59 I'll leverage some of our more Ops minded core devs at the nova one as well and I may know some folks in the loop on the glance mid cycle 16:13:07 let me assign myself a follow-up action 16:13:37 #action VW_ follow up with RS folks that might know about/be attending Glance mid cycle 16:14:22 Also, I bugged Tom about the Ops mid cycle a few days ago. Sounds like details are still pending 16:14:37 yeah that's another we should try to attend. 16:15:40 Ok - well it sounds like there is plenty of follow-up with regard to mid cycles. Anything else before we move on? 16:16:37 I'll go with no :) 16:16:48 #topic Periodic Tasks 16:17:22 So one of the things we discussed as a "quick win" in Paris was providing collective feedback on periodic tasks we regularly tweak/disable 16:17:32 did anyone bring some examples with them? 16:17:53 I did not, we're not currently tweaking them at all. 16:18:19 well we are disabling: heal_instance_info_cache_interval 16:18:23 only one i can think of off the top of my head is the intance info cache update interval 16:19:27 the main reason for us is our infrastructure design 16:19:58 it can be really heavy in the nova-network nodes, and we chose to have few of them... 16:20:06 I think we adjust that one too, belmoreira, but I'm admittedly behind in my homework 16:20:20 same for mdorman's 16:20:37 rabbit queues can get messy with a lot of the every minute updates 16:21:24 mdorman: can you elaborate on what you change and why with respect to that one? 16:22:03 i think we change it to 15 minutes or something. and tbh we only adjusted it because we heard others had issues with it, and we wanted to try to address it before it became a problem 16:22:18 fair enough 16:22:23 but yeah, intuitively, 60s updates on 100x VMs, is not such a great thing. 16:22:37 s/100x/100s/ 16:23:31 tangent - I know belmoreira uses cells. Do you mdorman? 16:24:00 yeah, that's interesting data. Cells vs no, and conductor vs no. 16:24:03 we've actually had to go to multiple global cells workers and a lot of the driver is the overall volume of updates from the computes 16:25:07 We are in the process of switching to cells. But currently we are not cells 16:25:08 we are working on converting to cells right now. but don’t have any prod/larger scale experience with them yet. 16:26:20 good to know klindgren / mdorman - happy to share insight as you move along. While not directly related to periodic tasks, I bring it up because of the reasons mentioned above 16:26:55 jlk - does condcutor make this whole line of conversation moot since a "smaller" number of hosts can send their task update info to a collection point? 16:26:56 We do use conductor though 16:27:38 For those looking into cells there is a working group for cellsv2. See: https://wiki.openstack.org/wiki/Meetings/NovaCellsv2#Agenda 16:28:23 really, klindgren - that's great - anything you can share relative to conductor and periodic tasks 16:28:39 thanks belmoreira - I keep trying to make that one and my schedule is not co-oprating 16:28:48 VW_: I honestly don't know. 16:29:01 VW_: conductor makes things nicer when it's computes directly writing to databases 16:29:08 but I don't think the periodic tasks are that. 16:29:09 going to put down and agenda item for next meeting to dive into current status of cells V2 for this group, belmoreira if you think it's worth it 16:29:20 good point, jlk 16:29:52 Not really - honestly we aren't at a point, yet, where we are big enough to run into some of these scaling issues. 16:29:53 yeah i would love more detials on cells v2, i too haven’t had the time/schedule to really get up to speed on it 16:30:36 VW_: definitely. I think this group should be informed and participate in the discussions 16:30:44 +1 16:31:14 Ok, so it sounds like we have a little feedback on periodic tasks. Perhaps take a summary of this chat to the mailing list? 16:31:23 +1 16:31:38 +1 16:31:48 is there a separate large deployers list or are we using the existing operators list? (prefer the latter) 16:32:04 the decision in Paris was use ops list but tag emails 16:32:12 nod 16:32:16 I also the: bandwidth_poll_interval 16:32:16 I've been using [Large Deployment Team] 16:32:25 at the moment the default is 600 16:32:33 ah yes -that one - how could I forget, belmoreira 16:32:55 but maybe it should be disabled... I don't see many people using it... 16:33:11 I tend to agree belmoreira 16:33:14 what'd be nice is a break down of A) what people alter, and B) what the impact of changing the default is 16:33:29 will work that into the email conversation 16:33:30 IE what purpose does that periodic task serve, and how is that purpose altered by adjusting the interval. 16:35:31 cool - we'll take that to the list then. maybe even back it with an ehterpad for better notes 16:35:50 moving along 16:35:59 #topic Meeting Times 16:36:30 In the docs: "Interval to pull network bandwidth usage info". However I don't know how useful it is, besides the queries that introduces... 16:37:03 right, that's kinda the problem. The definition is somewhat circular and doesn't describe what that data is USED for. 16:37:18 I'd like to lock down how we manage this meeting on an on-going basis. We agreed in Paris that once a month seemed right. With only one under our belt, I'm not ready to change that, but figured we can start there 16:37:23 noted belmoreira, jlk 16:37:30 jlk: yes 16:37:55 VW_: once a month seems about right. The more regular operators meeting can be used for whatever else comes up. 16:38:02 +1 16:38:09 agreed 16:38:30 speaking of which - did any of you make that yesterday? I had stuff at home I had to deal with 16:38:40 I did 16:39:05 Not to get too off-topic, but anything substantial worth revisiting here, klindgren 16:39:23 mainly around some packaging - then some other stuff that I am totally drawing a blank on 16:39:35 :-) 16:39:51 no worries 16:39:53 and the sample config file topic, which is loosely related to packaging 16:40:06 also we were trying to figure out if/when there’s a mid-cycle ops meetup 16:40:21 but i think mfisch/fifieldt are working on that one 16:40:28 yeah, that seems to be the burning question. Based on what I've heard, it's yes 16:40:33 just when and where is still in flight 16:40:42 but I'll try to catch Tom this evening 16:40:51 that’s my understanding too. there was an action to post curent status on all that to the ML 16:41:06 cool 16:41:58 back to our meeting is 3rd Thursday as good as anything for you all? 16:42:07 works for me 16:42:13 I picked this one to chat prior to holidays, but I don't see why we don't stick to it 16:42:23 works well for me, that's typically my work from home day 16:43:00 awesome 16:43:17 I'd like to flip the time every other month though so we can catch more timezones 16:43:27 anyone have serious concerns with that? 16:43:32 nope 16:43:55 no 16:43:56 sorry, I'm lost... 3rd Thursday? 16:44:23 I understand now... :) 16:44:23 3rd thursday of the month 16:44:30 yes, sorry for not being clear, belmoreira 16:44:39 sorry, quite late! 16:44:57 no problem, emagana 16:45:03 welcome 16:45:26 VW_: Representing Workday! We have a huge deployment going on! 16:45:56 VW_: oh there was one thing from the operators meeting yesterday 16:46:18 awesome - glad you could join us. we were just finalizing the structure for on-going meetings, emagana 16:46:24 go for it jlk 16:46:39 VW_: a call for deployers to write up SuperUser stories to describe how we each deal with code, how we get it from upstream, how we add any downstream changes, and how we turn it into something we put out into (pre)production 16:46:51 A longer format show and tell 16:47:22 While we don't think we can reach convergence on ways to do this, the community would benefit from having more explicit examples put out there, particularly when some are done using open source software 16:48:18 jlk: completely agree. 16:48:19 very nice - and noted, jlk - we also agreed in Paris that this group should get active in SuperUser. 16:48:59 and I agree with your assessment, jlk - but you probably knew that ;) 16:49:17 I just assume everybody always agrees with me 16:49:38 lol 16:49:45 so this time slot was pretty popular. We'll keep it as the Eurpoean friendly meeting time every other month starting in Feb. 16:50:10 and I'll send out a doodle to pick a time in the Evening US that brings in more of APAC 16:50:52 try to make so that if belmoreira is up smashing atoms late into the night he might be able to join as well :) 16:51:20 VW_: yeah :) 16:51:57 ok - cool. will get that out asap then so I can get stuff locked down in the wiki's etc 16:52:31 and not have to rely as much on email - which emagana may have pointed out I goofed up in etherpad notes 16:53:23 we are running low on time - which brings up my last question for this meeting schedule on-gong 16:53:32 s/on-gong/on-going 16:53:35 is an hour enough? 16:54:04 i think so. and if not, we should cap it 16:54:25 agreed 16:54:26 we all have enough meetings as it is 16:54:31 mdorman: +1 16:54:38 and these can be pretty efficient 16:55:01 sounds good. In theory, I like us having a bit of pressure to take unresolved/longer discussions to the list anyway - especially since we will have split meeting times with varying attendance 16:55:17 ok - cool - it stays at an hour then 16:55:48 ok - I want to honor at least one other agenda item before we run out of time 16:55:55 #topic Blueprints 16:56:08 anyone have any they would like to bring to the attention of the group? 16:56:32 I did not do good homework here. 16:57:00 any important ones with cells v2 we should read over, belmoreira? 16:57:54 There are some... let's see... 16:58:30 https://review.openstack.org/#/c/136490/1 16:58:46 #action VW_ send doodle out for alternative meeting time to support APAC members 16:59:00 this is for cells migration 16:59:04 #action VW_ get meeting details into wiki, etc now that it's been decided 16:59:41 got it - thanks, belmoreira 17:00:06 this etherpad about scheduling requirements is also important: https://etherpad.openstack.org/p/nova-cells-scheduling-requirements 17:00:24 looks like we are at time. if you have any more, belmoreira add them here - https://etherpad.openstack.org/p/Large-Deployment-Team-Meetings and we'll draw attention on the mailing list 17:00:50 thank you all for joining today! It was a little bumpy, but I'm happy with our first attempt 17:00:58 feel free to bounce me any feedback you ahve 17:01:01 thanks VW_ 17:01:14 thanks 17:01:15 thanks 17:01:24 I do have a question for people using ceilometer at scale 17:01:35 have you noticed that it hammers the hell out of the nova api? 17:01:41 I'll grab all the notes and get a summary out 17:01:51 #endmeeting