17:01:43 <Kiall> #startmeeting designate
17:01:44 <openstack> Meeting started Wed Aug 13 17:01:43 2014 UTC and is due to finish in 60 minutes.  The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:49 <openstack> The meeting name has been set to 'designate'
17:01:54 <Kiall> Heya - Who's here?
17:02:00 <timsim> o/
17:02:03 <eankutse> o/
17:02:03 <mugsie> o/
17:02:08 <rjrjr_> here
17:02:12 <vinod1> o/
17:02:46 <Kiall> Cool - Let's get started then :)
17:03:11 <Kiall> No actions from the last mee
17:03:12 <Kiall> t
17:03:31 <Kiall> #topic Release Status
17:03:37 <Kiall> #link https://launchpad.net/designate/+milestone/juno-3
17:03:55 <Kiall> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule
17:04:02 <Kiall> j3 is Sep 4th
17:04:04 <richm> here
17:04:29 <Kiall> I think the BP/Bug list @ https://launchpad.net/designate/+milestone/juno-3 is accurate and up to date
17:04:43 <Kiall> Other than Alex's "DNSupdate API - simplified DDNS update" BP
17:04:46 <timsim> Looks like pretty good progress has been made.
17:04:59 <Kiall> I've not seen anything on that one, anyone know anything about it?
17:05:07 <Kiall> if not - I'll bump.
17:05:16 <mugsie> [MiniDNS] Extend Designate's TSIG Key support to allow scoped TSIG keys <-
17:05:42 <mugsie> what the status with that one?
17:05:46 <mugsie> is it in for j3?
17:05:58 <Kiall> It should be done in time, I'll be doing it this week.
17:06:05 <mugsie> kk
17:06:36 <Kiall> Any other BPs etc not listed that should be? Lauchpad is awful for finding stuff.
17:06:59 <mugsie> mdns-designate-mdns-axfr ?
17:07:15 <Kiall> That's listed
17:07:20 <Kiall> https://blueprints.launchpad.net/designate/+spec/mdns-designate-mdns-axfr
17:07:30 <vinod1> https://blueprints.launchpad.net/designate/+spec/mdns-designate-mdns-tsig is not listed for j3
17:07:54 <Kiall> ah - nice catch vinod1 - that should be in too
17:08:09 <Kiall> done.
17:09:06 <timsim> Reckon server pools bps will be for the next one?
17:09:26 <mugsie> I think some could be alright
17:09:44 <mugsie> but... Openstack convention is not to do feature work past j3 riught?
17:10:02 <Kiall> Anything we can land without 1) breaking juno or 2) causing too much juno feature distraction is likely OK.. But I expect many of the BP and pool reviews will land in kilo
17:10:29 <Kiall> Convention says no new features from 3 weeks from now
17:10:34 <timsim> Probably good then.
17:10:58 <vinod1> are we creating a new branch after j3 for server pools etc?
17:11:06 <mugsie> thoughts on a feature branch for it? I have no problem maintaining it
17:11:10 <Kiall> vinod1: need to talk to infra re that.. maybe..
17:11:17 <mugsie> (aka rebasing master etc)
17:11:34 <mugsie> you want to action me or you to talk to -infra?
17:11:39 <Kiall> Part of the reason it's not the done thing is it's a distraction from stabilizing the release etc
17:11:43 <mugsie> Kiall: ^
17:11:50 <Kiall> I will,
17:12:00 <Kiall> #action kiall to talk to infra re SP branch...
17:12:32 <Kiall> Okay - Shall we move on and come back to ^ next week?
17:13:04 <Kiall> Taking silence as a yes :)
17:13:05 <Kiall> #topic Server Pools specifications
17:13:06 <timsim> Sure.
17:13:19 <Kiall> #link https://review.openstack.org/112688 - Server Pools - MiniDNS support
17:13:30 <Kiall> #link https://review.openstack.org/113462 - Server Pools - Manager
17:13:42 <Kiall> #link https://review.openstack.org/113447 - Server Pools - Storage
17:13:58 <Kiall> Those are the 3 I've noticed land - did I miss any?
17:14:06 <vinod1> Nope - that's it
17:14:06 <Kiall> s/land/be sumitted/
17:14:15 <mugsie> nope - I have an updated one to upload later on today
17:14:20 <Kiall> Who's had time to read them all?
17:14:30 <eankutse> I've read one
17:14:37 <rjrjr_> i have.  8^)
17:14:40 <eankutse> I am working towards the others
17:14:52 <vinod1> i have
17:14:53 <mugsie> not yet
17:14:54 <timsim> I haven't :x
17:15:08 <vinod1> though not the latest changes to pools manager
17:15:21 <Kiall> Okay.. Do the authors have anything in particular they want to discuss re them?
17:16:03 <rjrjr_> i just wanted to let people know they are there and to start reviewing them.
17:16:22 <Kiall> rjrjr_: makes sense :)
17:16:27 <vinod1> Re: https://review.openstack.org/112688 - Server Pools - MiniDNS support
17:16:29 <Kiall> Q - What BPs are still outstanding?
17:16:29 <rjrjr_> is there a way to look at them in a web page even though they are in review?
17:16:43 <rjrjr_> API is still outstanding
17:16:43 <mugsie> rjrjr_: yeah - the docs build job link
17:16:45 <vinod1> #link http://docs-draft.openstack.org/47/113447/3/check/gate-designate-specs-docs/4afac22/doc/build/html/specs/juno/server-pools-storage.html
17:16:51 <Kiall> rjrjr_: yes, use the -docs link posted by Jenkins after the build is done
17:17:10 <Kiall> Who's on point for the API one, was that you mugsie?
17:17:17 <rjrjr_> also, i don't have permissions to update the original BP link in launchpad.
17:17:23 <rjrjr_> API was becky
17:17:38 <vinod1> you mean Betsy?
17:17:46 <Kiall> rjrjr_: really? Ugh. Launchpad is a PITA
17:17:56 <rjrjr_> ^original BP link = specifications link
17:18:00 <vinod1> Betsy is off this week
17:18:41 <Kiall> Okay, well, should we review what we have this week, but hold off merging any until we have all the pieces in place etc
17:19:04 <mugsie> yup
17:19:05 <rjrjr_> yes, betsy.  sorry ...
17:19:12 <vinod1> Re: Server Pools - MiniDNS support, I have put some alternatives and reasonings under "Design considerations"
17:19:17 <Kiall> #action all to review the SP blueprints while the mid-cycle is still fresh.
17:19:30 <vinod1> rjrjr_: I had one question
17:19:47 <vinod1> Where/How would the pool manager know of the nameservers?
17:20:00 <rjrjr_> it reads them from the configuration file
17:20:01 <vinod1> Would that be covered in one of your specs?
17:20:13 <rjrjr_> name servers is from the servers table
17:20:43 <Kiall> Yes, for initial V1, the list of nameservers should be read from config, keeping things simple for now.
17:21:04 <rjrjr_> the difficulty with writing that spec.  are you talking backend DNS servers (the ones in config) or NS records?
17:21:20 <vinod1> the backend DNS servers
17:21:25 <rjrjr_> config file
17:21:53 <mugsie> config file - that should be part of the -manager spec
17:22:15 <Kiall> Total side note.. But if we can't keep the names straight in our heads, we need to fix it before exposing to users..
17:22:15 <rjrjr_> i have it in the storage spec, but i'll move it to be more clear.
17:22:52 <mugsie> rjrjr_: cool
17:23:01 <rjrjr_> i had a difficult time with it.  i know we have NS records (servers table) and servers.   very hard to write the spec with that clash.
17:23:05 <mugsie> I will add a terminology section to the overall spec as well
17:23:17 <mugsie> and we can then agree on the lexicon
17:23:19 <Kiall> mugsie: good idea
17:23:26 <rjrjr_> i kept hoping betsy's record table changes were done.
17:23:53 <rjrjr_> it has a ns_records table which should replace the servers table.
17:24:10 <mugsie> we can bump that up as a requirement, if needs be]
17:24:18 <Kiall> rjrjr_: I'm not sure it would, since end users can create NS records which we don't manage etc
17:25:03 <rjrjr_> are NS records really an attribute of a pool?
17:25:14 <rjrjr_> we have them now, but ...
17:25:33 <Kiall> Yes, they become an attr of the pool if memory serves me...
17:26:04 <Kiall> #action mugsie to write up SP terminology section in the main SP blueprint.
17:26:09 <rjrjr_> what is the purpose of having them there versus just creating them in the zone?
17:26:17 <rjrjr_> like we do with other records?
17:26:35 <rjrjr_> the only reason to have it in the pool is because one is required for a zone.
17:27:16 <Kiall> Well, they will exist in both places.. as NS records in the `records` (or `ns_records`) table, and those will be kept up to date with the the values for the pool.
17:27:44 <rjrjr_> that is what i had assumed
17:28:06 <mugsie> but we would have to store the inital pool ns records somewhere as well
17:28:13 <Kiall> mugsie: exactly :)
17:28:31 <rjrjr_> ns_records table with a pool_id column in that table.
17:29:28 <eankutse> that sounds reasonable to me
17:29:32 <Kiall> probably managed = True, managed_type = pool, managed_id = $pool_id etc - matching what we do elsewhere
17:29:44 <mugsie> yeah... that ^ could work
17:30:09 <Kiall> Okay .. Let's move on? A few actions above, and a few BP's to read + review before next week.
17:30:46 <Kiall> #topic Follow-up discussion on Database/MiniDNS scalability
17:30:50 <Kiall> #link https://wiki.openstack.org/wiki/Designate/MdnsScalability
17:31:12 <Kiall> we had a quick chat re timsim's concerns in #openstack-dns yesterday if anyone feels like reading the scrollback
17:31:18 <timsim> Essentially our folks are worried about MiniDNS relying on/reading from a database many times that could be far away, and whose connection could die periodically. They're not convinced at all about replicating MySQL on the scale we'd need.
17:31:19 <Kiall> timsim: want to recap the concerns?
17:31:46 <timsim> We had a couple of possible solutions, or mitigations, to this issue that I posted on that page.
17:32:13 <eankutse> ^ #link https://wiki.openstack.org/wiki/Designate/MdnsScalability
17:32:21 <timsim> Namely reimplementing the agent as a sort of reflection of MiniDNS that sat on a Bind9 Master, that would allow Bind9 to maintain a traditional Master-slave set up. This would just be another pool manager plugin for Bind9, apart from the default one.
17:33:25 <timsim> There was also talk of implementing some sort of NoSQL storage backend.
17:33:43 <timsim> Or Caching at the MiniDNS level.
17:34:08 <mugsie> yeah - I think we should get the AFXR support merged, and the do real life load testing
17:34:20 <mugsie> across multiple continents
17:34:30 <Kiall> From the discussion yesterday, We (I anyway!) came to the conclusion that A) timsim's concerns are valid (just putting it out there..) and B) We have a bunch of ways to fix it, some in mDNS directly (caching..) and some external to mDNS etc, which can be implemented in isolation.
17:35:02 <richm> How is scaling the mDNS database different than scaling any other MySQL/NoSQL application?
17:35:03 <Kiall> mugsie: yep - Once we have AXFR merged, we can really start testing for scale issues
17:35:36 <mugsie> personally I am not sure how Option 1 is any better that async MySQL replication - Cassandra / Mongo have to send the same data the same distance - they are also eventually consistant
17:35:37 <rjrjr_> is one designate database necessary are can we break things up by region, pool, etc.?
17:35:46 <timsim> richm: Specifically it's replicating the entire Designate database.
17:35:55 <Kiall> richm: I personally agree with that, and think NoSQL is probably not the best option
17:36:42 <rjrjr_> say these Designate instances manage these pools these servers these zones have one database.  these other Designate instnaces manage these other pools other servers other zones on another database.
17:36:45 <mugsie> but I would think that we would get a lot better of an idea with real world replictaion tests
17:36:46 <timsim> DandyPandy: If you're around, could you talk briefly about the issues of scaling MySQL that far?
17:36:58 <DandyPandy> Sure
17:37:11 <mugsie> rjrjr_: the data still has to get from say, Austin -> Sydney
17:37:16 <timsim> DandyPandy is one of our Ops guys.
17:37:20 <mugsie> o/
17:37:34 <DandyPandy> MySQL master/slave replication is prone to breaking and there are no good way to cleanly recover that replication in an automated fashion.
17:37:59 <DandyPandy> Using Galera is horrible over highly latent connections since writes are done synchronously.
17:38:01 <rjrjr_> you distribute the data like we distribute DNS zones in a typical DNS setup.
17:38:24 <eankutse> I hear that the MuSQL replication gets even worse if you go past one slave
17:38:35 <timsim> rjrjr_: That's fine if you don't need the same pools replicated in two different places.
17:38:51 <Kiall> Yep - Galera over WAN == Pain, but one way MySQL replication cross continent isn't something I've heard Ops folks say is broken before
17:39:23 <mugsie> yeah - tbh, I know master -> slave was an issue in the past, but I thought it was solved
17:39:40 <DandyPandy> Kiall: The problem is that it requires manual intervention when it does fail.
17:39:57 <richm> I guess I don't understand - is there something different about designate/dns that makes the db scaling/high availability/wan problem much harder?  I thought this was a pretty standard deployment configuration for distributed databases.
17:40:16 <DandyPandy> mysql isn't a distributed database
17:40:18 <richm> That is - designate cannot be the first application to have faced this problem
17:40:53 <mugsie> ok - so its a MySQL specific problem
17:40:54 <Kiall> DandyPandy: Yes, MySQL has no automated recovery built in, but lots of tooling exists to automate etc.. Anyway.. I guess this is all besides the point, global DB replication is something that we should probably work to make optional if at all possible.
17:41:09 <mugsie> how about other SQL DBs?
17:41:13 <Kiall> (i.e. I agree, we should look for + find alternatives)_
17:41:37 <richm> I doubt that designate is even the first openstack component to have faced this problem
17:41:39 <DandyPandy> richm: I think most people use API's rather than direct db connections
17:41:54 <Kiall> richm: the issue is, once all the DNS servers start hitting the DB, the query load will be much higher.
17:42:26 <mugsie> yeah - but the miniDNS server queries will be read only
17:42:28 <Kiall> with millions of zones and a short refresh internal, and N masters listed on each backend DNS server, the #'s get pretty large
17:42:46 <eankutse> ++
17:42:58 <DandyPandy> something that could be eventually-consistent (such as cassandra), would be awesome, but I understand that isn't something openstack is particularly open to
17:43:14 <mugsie> And, I think we do need to improve caching in any case (even excluing multi continental links)
17:43:18 <Kiall> mugsie: Yes, which read only MySQL slaves essentially solves .. But it requires global mysql slaves, which DandyPandy etc are hoping to avoid..
17:43:50 <Kiall> DandyPandy: Yea, OpenStack as a whole is mostly a MySQL shop.. even when we claim support for others ;)
17:44:01 <mugsie> yeah ... but any of the solutions will most likely require a DB slave of some sort in each POP
17:44:09 <DandyPandy> Kiall: BUT SQLALCHEMY SUPPORTS ORACLE!!!
17:44:22 <timsim> Problem solved ^
17:44:32 <Kiall> DandyPandy: I wouldn't shout that too loud.. The SQLA author is now working on OpenStack for RedHat ;)
17:44:42 <Kiall> (He's probably in the room right now ;))
17:45:33 <mugsie> :)
17:45:57 <mugsie> Can we agree to come back to the is with real data?
17:46:05 <Kiall> Anyway .. I think we should figure out if there are any reasonable alternatives to global MySQL read only slaves, some ideas are in the wiki page timsim put together.. Can we all have a read and a think, then come back to this?
17:46:14 <mugsie> +1
17:46:19 <rjrjr_> +1
17:46:29 <timsim> Kiall: +1 i can add some thoughts on caching as well.
17:46:35 <eankutse> +
17:46:38 <timsim> Maybe we chat on it again next week?
17:47:06 <mugsie> cool
17:47:14 <rjrjr_> seems like a solution should align with pools since that is the reason we are using pools.
17:47:34 <Kiall> I also asked timsim to try get some real life numbers, even if they are just distributions of query types/sizes/etc rather than absolute numbers (I can see absolute numbers being hard to release..) at least then, we'll have a better idea of where to work on first
17:47:37 <mugsie> rjrjr_: I think this will be a 'optional add on'
17:47:47 <timsim> Kiall: Working on that ;)
17:48:01 <mugsie> 95% use case will be the standard one we have laid out
17:48:10 <timsim> mugsie: agree.
17:48:18 <Kiall> agreed with mugsie there, We need to be careful to keep things simple for tiny deploys, while also supporting large scale
17:48:29 <mugsie> this will be for the top 5%, or even 1% (or even...  single) deployments
17:48:36 <timsim> The issues we're talking about are for really big deployments.
17:48:48 <eankutse> mugsie: :-)
17:48:52 <mugsie> but it does deserve carefull consideration
17:48:55 <Kiall> #action all to review https://wiki.openstack.org/wiki/Designate/MdnsScalability
17:49:02 <Kiall> #action kiall to put MdnsScalability back on agenda for next week.
17:49:53 <Kiall> Anything else on this before we move to Open Discussion?
17:49:57 <timsim> DandyPandy: Thanks for stopping by
17:50:09 <DandyPandy> timsim: not a problem!
17:50:14 <Kiall> DandyPandy: yep - thanks, hopefully we can have a more involved conversation next week ;)
17:50:43 <Kiall> #topic Open Discussion
17:50:51 <Kiall> Okay - Anyone have any off-agenda items?
17:50:55 <mugsie> yup - 1
17:51:04 <mugsie> the review list is getting.... long
17:51:17 <mugsie> https://review.openstack.org/#/q/status:open+project:openstack/designate,n,z
17:51:38 <vinod1> I will take a look at some of them later today
17:51:56 <mugsie> just to keep on top of it
17:52:28 <Kiall> Yea, with mugsie + myself away for the last 2.5 weeks, betsy away this week, the list has grown ;)
17:52:39 <timsim> mugsie: Somewhat unrelated, is it possible to share that dashboard you showed at the mid-cycle?
17:52:48 <mugsie> timsim: yup
17:52:53 <mugsie> will put a link on the wiki
17:52:57 <Kiall> So not gonna work...
17:52:58 <mugsie> (it is quite lng)
17:52:58 <Kiall> https://review.openstack.org/#/dashboard/?foreach=%28project%3Aopenstack%2Fdesignate+OR+project%3Aopenstack%2Fpython-designateclient+OR+project%3Aopenstack%2Fdesignate-specs%29+status%3Aopen+NOT+owner%3Aself+NOT+label%3AWorkflow%3C%3D-1+label%3AVerified%3E%3D1%252cjenkins+NOT+label%3ACode-Review%3E%3D-2%252cself+branch%3Amaster&title=Designate+Review+Inbox+%28master+branch+only%29&Needs+Feedback+%28Changes+older+than+5+days+that+have+not
17:52:58 <Kiall> +been+reviewed+by+anyone%29=NOT+label%3ACode-Review%3C%3D2+age%3A5d&Your+are+a+reviewer%2C+but+haven%27t+voted+in+the+current+revision=reviewer%3Aself&Needs+final+%2B2=label%3ACode-Review%3E%3D2+limit%3A50&Passed+Jenkins%2C+No+Negative+Feedback=NOT+label%3ACode-Review%3E%3D2+NOT+label%3ACode-Review%3C%3D-1+limit%3A50&Wayward+Changes+%28Changes+with+no+code+review+in+the+last+2days%29=NOT+label%3ACode-Review%3C%3D2+age%3A2d&Specs+in+need+
17:52:59 <Kiall> of+review=project%3Aopenstack%2Fdesignate-specs
17:53:11 <timsim> lol
17:53:15 <mugsie> i knew ^ wouldnt work ;)
17:53:27 <Kiall> http://paste.openstack.org/show/94532/
17:53:36 <eankutse> :-)
17:53:44 <mugsie> even if non cores had a look, it makes core reviewing easier
17:53:53 <timsim> Yeah that's cool.
17:54:08 <Kiall> Okay - 5 mins before we get kicked out by Trove, anything else?
17:54:19 <eankutse> Cool stuff
17:55:03 <Kiall> Going once....
17:55:19 <mugsie> link is here for futre reference https://wiki.openstack.org/wiki/Designate#Review_Dashboard
17:55:52 <Kiall> Okay, let's call it so, thanks all :)
17:55:56 <mugsie> o/
17:55:59 <eankutse> thx
17:56:01 <Kiall> #endmeeting