03:00:25 <VW_> #startmeeting Large Deployments Team September 2015 Meeting
03:00:26 <openstack> Meeting started Fri Sep 18 03:00:25 2015 UTC and is due to finish in 60 minutes.  The chair is VW_. Information about MeetBot at http://wiki.debian.org/MeetBot.
03:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
03:00:29 <openstack> The meeting name has been set to 'large_deployments_team_september_2015_meeting'
03:01:18 <VW_> alrighty, let's kick this thing off.  Lose agenda still here - https://wiki.openstack.org/wiki/Meetings/LDT#Meeting_Times
03:01:55 <mdorman> who’s all here?
03:02:08 <klindgren> o/
03:02:13 <VW_> o/
03:02:36 <klindgren> sorrison, arrives just in time
03:02:42 <mdorman> heh
03:02:51 <sorrison> hey guys
03:02:55 <mdorman> at least i am staying up late for a goo d reason then :)
03:03:03 <VW_> hee hee
03:03:08 <mdorman> “late"
03:03:17 <sorrison> was just at lunch!, curry Friday
03:03:22 <mdorman> :)
03:03:57 <sorrison> also fighting fires with our kilo upgrade and migrating our pre prod cloud to neutron....
03:04:04 <mdorman> so what were the business logic patches?   i think i missed that part at the meetup
03:04:19 <VW_> #topic Business Logic Patches
03:04:23 <mdorman> sorrison:  just a day in the life, yeah?
03:04:35 <VW_> think it actually game from the meeting the following week, mdorman
03:04:43 * VW_ is looking back over the logs
03:04:57 * VW_ was in a keynote at openstacksv at the same time as said meeting
03:05:17 <mdorman> ah ok
03:05:20 <mdorman> yeah i missed that one too
03:06:15 <klindgren> I htink business logic patches are patches that we are carrying to implement business logic
03:06:20 <VW_> 16:49:01 <klindgren> I plan on starting another etherpad of patches that we have for just business logic
03:06:55 <mdorman> ok yeah, so just trying to gather a list.  i get it.
03:06:58 <klindgren> hopefully a session in Tokyo dedicated to it.  So I still need to createa an ehter pad for that
03:07:19 <klindgren> INclusive of all ops, try to see whats out there/overlap ect ect
03:07:21 <mdorman> klindgren:  had you dropped that in to the tokyo ops track ehterpad as a suggested session?
03:07:34 <klindgren> and see if we can't get some traction on getting stuff into openstack
03:07:37 <mdorman> i think this is a great idea… i’m sure we can find a lot of commonality there too
03:07:40 <klindgren> or atleast hooks/extension points
03:07:52 <klindgren> it is in the tokyo ehterpad
03:08:06 <klindgren> and I asked for it to be a working group as well
03:08:09 <klindgren> but we shall see
03:08:45 <mdorman> cool
03:09:02 <VW_> be careful what you asked for klindgren
03:09:05 <VW_> :P
03:09:12 <mdorman> you’re gonna end up moderating a ton of stuff :)
03:09:18 <mdorman> so, an action for kris to set up an etherpad for that?
03:09:19 <VW_> you might be in charge of something before too long
03:09:33 <VW_> #action klindgren set up business logic etherpad
03:09:38 <VW_> it is now
03:09:41 <mdorman> haha
03:09:44 <klindgren> I will see if I can avoid that for now :-)
03:09:46 <klindgren> hehehe
03:10:29 <klindgren> Anyway - I plan on trying to get that up next week
03:10:43 <klindgren> and send a mailing list out to ops to update with what they have
03:11:03 <mdorman> cool
03:11:08 <VW_> sounds good to me
03:11:57 <VW_> moving on then, I suppose
03:12:07 <klindgren> was going to say - I have nothing else on that
03:12:23 <klindgren> unless fifieldt has some comments :-)
03:12:33 * fifieldt looks up
03:12:40 <mdorman> haha
03:13:11 <VW_> uh oh - didn't know he was lurking
03:13:26 <VW_> #topic Cells V1
03:13:38 <VW_> so, I'll be the first to admit I'm behind on this one
03:13:59 <klindgren> Yea - though I sent an email to ops today
03:14:00 <fifieldt> re: TYO ops, etherpad should be become agenda very soon now :)
03:14:09 <klindgren> outlining the stuff that was specifically broken in cells
03:14:19 <mdorman> so there is https://review.openstack.org/#/c/215459/2 for some of the v1 pathes we’ve been carrying
03:14:37 <mdorman> i kinda just threw it out there as best i could, and it is getting some reviews.  folks are asking for more detailed unit tests, and i could use some help on that.
03:14:39 <klindgren> that has a pretty good list of all the stuff that we are borrowing from nectar
03:15:14 <klindgren> The unit test stuff is always a little frustrating to me
03:15:22 <mdorman> hoping to maybe get mgagne or others to lend a hand with that.  i barely understand how the code is working, much less how to build a good unit test
03:15:30 <klindgren> especially on cells - its like you broke it and never tested it the first time around...
03:15:48 <mdorman> yeah.  i mean it’s not an unreasonable ask, imo
03:15:51 <klindgren> but - getting good tests in place would be nice
03:16:09 <VW_> gotcha
03:16:09 <VW_> will take a look
03:17:25 <klindgren> anyway - I was going to take my email and update the ether pad with the cells patches
03:17:39 <klindgren> 99% comes from NeCTAR anyway
03:17:54 <VW_> yeah.  Truth be told, I'd like to have a good hear to hear with Garbutt and Laski about cells v1 and what we can do in the next couple of cycles
03:18:01 <VW_> heart to heart even
03:18:12 * VW_ inspects his t key
03:19:01 <klindgren> If anything just having a list called out with links to code - could help people who need to move to cells and have stuff work
03:19:10 <VW_> yeah
03:19:12 <mdorman> right.
03:19:27 <mdorman> i think we’re going in a positive direction, though.  this issue and the patches are at least getting some attention
03:19:28 <sorrison> that interface attach patch is still under review right?
03:19:44 <klindgren> becuase its like hey I am testing out cells and - surprise - you know a bunch of stuff that you use to do... yea - thats not fixed/working
03:20:02 <klindgren> sorrison, yea - they want more unit tests
03:20:13 <klindgren> I think is the main complaint that I saw
03:20:57 <mdorman> sorrison:  yup.  https://review.openstack.org/#/c/215459/2
03:21:11 <VW_> klindgren: will you link the etherpad again
03:21:11 <mdorman> i have been meaning to make some trivial updates and post another patchset, but just have not had time.
03:21:19 <VW_> just in case folks don't read last months notes
03:21:36 <mdorman> #link cells v1 interface attach patch:  https://review.openstack.org/#/c/215459/2  could use some help in implementing better unit tests
03:22:35 <klindgren> #link Cells patches etherpad https://etherpad.openstack.org/p/PAO-LDT-cells-patches
03:23:20 <VW_> so, I would suggest that in the ehterpad, we start making notes of what we need for each patch
03:23:37 <VW_> like "could use some help in implementing better unit tests"
03:24:15 <serverascode> (o/ here but late)
03:24:35 <VW_> welcome serverascode
03:24:49 <VW_> we were just chatting about Cells V1 patches
03:25:04 <VW_> we are trying to get a good list of common ones that we can advocate for with the Nova folks
03:25:11 <serverascode> cool thanks
03:25:29 <VW_> sounds like this may be a big topic of discussion in LDT session at Tokyo
03:26:38 <VW_> would you all agree?
03:26:41 <klindgren> yea
03:26:46 <klindgren> I think so
03:26:48 <VW_> cool
03:26:52 <mdorman> +1
03:27:34 <VW_> ok, well, on that note...
03:27:43 <VW_> #topic Tokyo
03:28:30 <klindgren> Are we also going to try to meetup with carl/neutron people on the network segmentation stuff or? - I would really like to not see that slip away
03:28:45 <VW_> that would be really good
03:28:52 <VW_> I'm happy to devote time to i
03:28:57 <VW_> to it
03:29:11 <VW_> I don't know how much time fifieldt thinks he can give us yet
03:29:27 <gfa> +1 to the network segmentation stuff
03:29:31 <fifieldt> LDT should get an 80min working group block I think
03:29:40 <VW_> hazzah!
03:29:58 <mdorman> maybe we can try to schedule part of that time to meet with neutron folks, kind of like we did at the midcycle meetup
03:30:10 <VW_> yeah - I think that makes a lot of sense
03:30:23 <VW_> 1.  Meet with Neutron
03:30:35 <VW_> 2.  Solidify cells V1 patche list
03:30:39 <mdorman> i know they are also working on finalizing thier own adgenda, so maybe it would make sense to put a plug in on that side and see if they’re willing to devote time to the topic too
03:30:54 <VW_> that is a good point
03:31:28 <VW_> Kyle is not re running as PTL either, so it might be tricky to figure out their schedule
03:31:35 <VW_> I have a core I can ping.
03:31:39 <klindgren> Yea - I would be OK trying to go to a neutron meeting about network segmentation - but in the past those meetings kinda devolved into something else
03:31:53 <klindgren> I think Carl is runnign for PTL as well
03:32:03 <VW_> klindgren - you want to reach out to carl
03:32:08 <klindgren> I will
03:32:35 <sorrison> I am using a patch with helps with dhcp agent scheduling
03:32:38 <VW_> #action klindgren check with Carl on possible Neutron session on segmentation
03:32:42 <klindgren> we have a talk accepted as an alternate on this very topic so I will reach out to him about that as well
03:33:02 <VW_> nice!
03:33:03 <sorrison> https://review.openstack.org/#/c/205631/ works well
03:33:42 <klindgren> Curious - in tokyo I think we have our hands full with stuff currently on the plate
03:33:49 <klindgren> but I would also like to talk about glance
03:34:04 <VW_> hrm
03:34:05 <klindgren> and getting some useful (basic imho) into glance
03:35:07 <klindgren> like quotas and hierarchical project support
03:35:15 <VW_> hey - so, we actually have a chance to meet before the summit
03:35:20 <sorrison> +1
03:35:29 <mdorman> thre has been some ML discussion about Mitaka priorities for glance, but i haven’t been following it really
03:35:30 <VW_> if we can confirm a Neutron session, maybe we can spend our time on Cells and Glance
03:35:51 <mdorman> seems like a good approach
03:37:07 <VW_> but I may also see if I can get some glance dev love in that meeting too
03:37:14 <klindgren> I dont want to sound like debbie downer, but I was talking to someone about glance - and what I heard was glance might need a new PTL before some of those features get implemented
03:37:29 <klindgren> but they are soooo very desperately needed
03:37:30 <VW_> #action VW_ see if a glance dev can join OCT ldt meeting
03:38:08 <VW_> I personally wich we could get glance out of path for data transfer
03:38:41 <klindgren> so have glance redirect nova to upload to swift or somethign directly
03:38:42 <VW_> use it to catalogue images and such, but not have to push bits through it
03:38:42 <klindgren> ?
03:38:55 <VW_> yeah - something like that
03:39:36 <klindgren> I heard an idea from someone a while back to run glance on every compute node
03:39:38 <VW_> Our friends at CERN  have cooked up some pretty impressive caching stuff to work around/with those constraints
03:39:41 <klindgren> without an image cache enabled
03:39:54 <klindgren> to effectively do that
03:40:26 <klindgren> yea - that would be pretty nice as well.  BUt I will set for jsut some admin basics
03:40:51 <sorrison> I went to the ops glance session in vancouver, they would love ops input. I was pretty much the only op there
03:40:52 <klindgren> like quotas, "cloud provided images", hierarchical projects
03:41:58 <klindgren> so VW_ you would also like a glance data bypass option?
03:42:21 <VW_> I would personally becuase of the amount of hardware I have to throw at the damn service
03:42:50 <klindgren> kk
03:42:59 <VW_> we were headed down that path, but stopped.  I need to dive back into that with our devs, but I've had my hands full with some other things
03:43:09 <klindgren> sorrison, thats good feedback
03:43:15 <VW_> well, at least we were looking at bypass for uploads
03:43:43 <gfa> you can do that glance on every compute to bypass it for uploads
03:43:51 <sorrison> I talked about bypass with them, took a while to explain to them with some crazy whiteboard drawings
03:43:59 <gfa> not bypass the service, but the load on the glance farm
03:44:40 <klindgren> yea - but I think VW_ also wans to avoid also running 20k+ instances of glance as well
03:44:52 <VW_> :)
03:45:09 <gfa> hahah lol, right
03:45:28 <klindgren> we can add that to the list
03:45:39 <VW_> for sure
03:45:50 <klindgren> so we want to also try to tackle some asks around glance then at Tokyo as well?
03:45:54 <VW_> and I'll see if we can get some Glance folks in for the Oct meeting ahead of the summit
03:45:57 <klindgren> kk
03:46:00 <klindgren> ah right right
03:46:03 <klindgren> Oct meeting
03:46:13 <VW_> ok, so before we run out of time, I have a bit of bad news
03:46:20 <VW_> and fifieldt is going to be mad at me
03:46:26 <fifieldt> :D
03:46:32 <VW_> but I don't think I'm going to be able to come to Tokyo after all
03:46:48 <VW_> running a big cloud may have to win that week
03:46:49 * fifieldt asplode
03:46:54 <mdorman> wah wah
03:47:08 <mdorman> bummer
03:47:33 <VW_> yeah - looks like I'll be heading the other way, actually and ending up in Lond that week for some DC migration stuff
03:48:02 <VW_> I chatted with klindgren a bit earlier.  He's willing to help moderate the session (I think)
03:48:13 <VW_> ;)
03:48:21 <sorrison> booo
03:48:28 <klindgren> yea I volunteered mdorman to help too :-)
03:48:33 <VW_> cool
03:48:43 <sorrison> happy to help out too
03:48:51 <klindgren> on the plus side I do hava a hotel for tokyo and my openstack ticket - now just need to book flights
03:49:01 <VW_> and I'll have two of my engineers there - Joel that came to Palo Alto and Ben that was in YVR
03:49:20 <VW_> thanks guys - really sorry about this
03:49:27 <mdorman> cool
03:49:32 <VW_> but it was a tough call I had to make
03:49:50 <VW_> if things change, I'll let you know in the Oct meeting
03:50:03 <VW_> and hopefully fifieldt will be speaking to me again by then
03:50:03 <klindgren> #link https://etherpad.openstack.org/p/LDT-glance-asks
03:50:11 <VW_> sweet!
03:51:09 <VW_> but, that brings up one thing that I was going to ask you all in Tokyo about the working group and how we manage it.
03:51:57 <VW_> I'm happy to keep co-ordinating things, but I didn't know if that was a role we felt we should move different people in to
03:52:21 <VW_> was just going to get thoughts while we were gathered
03:53:17 <klindgren> I think as long as it doesn't always fall on one person and people volunteer to leader/co-ordinate certain things it shouldn't be a huge issue
03:53:28 <mdorman> i’m happywith the way things are now.  but if it’s something where you’re feeling you don’t have the time to manage it, then it’s fair to ask someone else to step up.
03:53:28 <klindgren> at least not right now
03:54:26 <VW_> I'm fine continuing to be the cat hearder if you guys see value in it
03:54:39 <klindgren> I kinda hoped that we would have more people participate in LDT
03:54:46 <VW_> I just didn't want it to be all "my" thing - or appear to be
03:54:52 <VW_> me too
03:55:04 <VW_> we are gaining some momentum - all be it, slowly
03:55:11 <mdorman> nah i don’t think it’s that at all.  having someone run the meetings and kinda keep things moving is good.
03:55:12 <klindgren> maybe we should call it the ADT :-)
03:55:18 <klindgren> All deployment team
03:55:22 <mdorman> haha
03:55:36 <klindgren> though the security company may not like that
03:55:56 <VW_> lolz
03:55:59 <serverascode> was there still going to be some time in tokyo for public clouds in the ldt stuff?
03:56:33 <VW_> yeah, trying to figure out how we cover it all in 80 minutes
03:56:49 <VW_> ideally we can be IN a Neutron discussion
03:57:02 <serverascode> ah ok 80 min total
03:57:04 <VW_> so that frees one thing of the list for our session
03:57:17 <klindgren> I am all for public cloud stuff in LDT
03:58:01 <VW_> so, let's do this then. We know that 4 things are on the ideal list for Tokyo
03:58:07 <VW_> 1.  Neutron Segmentation
03:58:17 <VW_> 2.  Cells v1 patches
03:58:24 <VW_> 3.  Glance
03:58:29 <VW_> 4.  Public Clouds
03:58:44 <klindgren> serverascode, anything burning in public cloud space
03:59:01 <klindgren> many ideas were brought up in the etherpad and at palo alto
03:59:06 <VW_> let's see what we can work out with the devs in the thre projects between now and the Oct meeting and then we can prioritize the 80 minutes in the next meeting
03:59:40 <klindgren> sounds good to me
04:00:02 <serverascode> I just know there are at least 13 public clouds based on openstack now, so I'd like to talk to those people somehow :) no major technical things right now
04:00:20 <mdorman> +1 to taht plan VW_
04:00:37 <VW_> fifieldt: LDT is happy to abosorb the public cloud stuff for now, but do you think it warrants a session of its own?
04:01:00 <klindgren> serverascode, are you including Godaddy in that list?
04:01:09 <fifieldt> personally, I think those public clouds should be working with LDT
04:01:29 <fifieldt> might not be the exactly right label, but I think you all have a similar enough mindset :)
04:01:30 <xavpaice> some public clouds aren't exactly 'large' though ;)
04:01:44 <serverascode> just going off this list: https://github.com/openstack/os-client-config/tree/master/os_client_config/vendors
04:01:46 <VW_> is there sufficient quantities of whiskey to talk you into a triple lenght session like YVR :)
04:02:23 <fifieldt> indeed xavpaice, but are their issues not being covered by the LDT? :)
04:02:25 <xavpaice> oh, didn't know about that - must add us on
04:02:44 <xavpaice> fifieldt: more that not all the LDT issues are relevant for tiny clouds
04:02:49 <klindgren> the problem is I have some other sessions that I want to go to as well
04:02:49 <xavpaice> rather than the opposite
04:03:20 <fifieldt> I'm just trying to work out whether there's a real issue here :) a gap, if you will :)
04:03:53 <xavpaice> I'd be keen to be involved in LDT if that is also public clouds - and I'd be extremely happy to learn from bigger players that aren't operating public clouds
04:04:02 <xavpaice> there's a lot of common ground
04:04:34 <mdorman> i think tim bell said it well in paris, LDT is for you if you consider yourself a large deployer.  which may or may not correspond to some number of cores, etc.
04:05:02 <VW_> I think so too, xavpaice.  For example, a "small" cloud may still chose to do something like break up capacity into different cells based on hardware types to offer customers different products
04:05:36 <VW_> plus, I don't know that any service providers/public clouds with dreams of staying small ;)
04:05:46 <VW_> s/with/have
04:05:47 <xavpaice> true enough
04:05:54 <serverascode> that is true for sure
04:06:08 <klindgren> ah - back in the days when we only had a few hundred vm's
04:06:11 <mdorman> any other topics?
04:06:13 <fifieldt> we can't let xavpaice go - he has too many cool stories about datacentre floods
04:06:30 <VW_> woah - it's settled then
04:06:52 <xavpaice> splishy splashy
04:07:07 <mdorman> liquid water cooled, DC?  nice!
04:07:28 <xavpaice> someone thought it a good idea to put the generator and UPS batteries in the basement
04:07:45 <klindgren> side question - dont hold the meeting open.... but did peoples kilo upgrades go smothly
04:07:58 <sorrison> we are in the middle of ours
04:08:02 * xavpaice would love to hear war stories there
04:08:05 <xavpaice> about to do it
04:08:05 <klindgren> I am hearing reports from a number of people of issues post update
04:08:08 <sorrison> done nova and been ok
04:08:21 <klindgren> like the ipset stuff in neutron
04:08:26 <klindgren> we hit that like 4-5 times
04:08:27 <VW_> I think so - we on a trunk pull from last month at the moment
04:08:30 <klindgren> then nohting for 2+ weeks
04:08:33 <VW_> working on another
04:08:36 <sorrison> but seeing increased load on keystone (juno) which in tern is hitting ulimits in memcached
04:09:04 <VW_> anything else "official" before we tiptoe down upgrade lane?
04:09:09 <mdorman> fernet tokens, sorrison ?
04:09:14 <mdorman> VW_:  not for me
04:09:21 <sorrison> also default_ephemeral_filesystem config changed I think
04:09:27 <klindgren> I dont have anything official
04:09:28 <VW_> I'll make sure the Oct meeting is all about piortiziing the tokyo schedule
04:09:42 <klindgren> kk
04:09:47 <sorrison> we had vms spinning up with fat filesystems for vdb...
04:09:49 <xavpaice> is this the normal time for LDT meetings?
04:09:54 <VW_> sorrison: I'll try to get something to you ahead of the meeting
04:10:01 <VW_> every other month, xavpaice
04:10:18 <sorrison> yeah looking into fernet, once we've got everything onto kilo,
04:10:30 <xavpaice> right - I happened in here by accident today, will try to be more organised in future
04:10:31 <sorrison> worried though as we use pki and so no verification on keystone server
04:10:31 <VW_> next month, it will be third thursday at 16:00 UTC
04:10:45 <sorrison> load on keystone once we verify using fernet
04:10:52 <xavpaice> that's 4am here in NZ - probably won't join
04:10:54 <VW_> we aren't the best record keepers, xavpaice, but https://wiki.openstack.org/wiki/Meetings/LDT#Meeting_Times
04:11:10 <xavpaice> many thanks
04:11:20 <VW_> sure thig
04:11:21 <sorrison> xavpaice where you based?
04:11:26 <xavpaice> New Zealand
04:11:31 <VW_> #endmeeting