17:01:19 #startmeeting Designate 17:01:20 Meeting started Wed Feb 4 17:01:19 2015 UTC and is due to finish in 60 minutes. The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:23 The meeting name has been set to 'designate' 17:01:42 Heya - Who's about? 17:01:44 o/ 17:01:44 o/ 17:01:48 o/ 17:02:15 o/ 17:03:08 No actions from last meet, so first topic straight to the first topic: 17:03:11 #topic Kilo Release Status (kiall - recurring) 17:03:17 ello 17:03:17 #link https://launchpad.net/designate/+milestone/kilo-2 17:03:55 We have lots of bugs marked as in progress, meaning code reviews are up.. k2 sha1 needs to be given to Thierry tomorrow - The rest of my day is code reviews, if others can do the same hopefully we can land a few of those. 17:04:19 Anything that looks risky to land, shouldn't be landed :) 17:04:57 Anyone have any reviews they think really need to land? 17:05:06 I think rjrjr_'s cache/pools one probably needs to 17:05:26 https://review.openstack.org/#/c/148386/ imo 17:05:50 Cool - betsy or mugsie will need to review or approve ^ one 17:06:20 Any others? 17:06:58 I'll take that as a no then ;) 17:07:05 Yeah just rjrjr_ 's 17:07:13 betsy is off today so mugsie needs to step in :-) 17:07:33 Hah - mugsie is out too, but attending the meet ;) 17:08:43 Okay - We'll see what we can land before tomorrow, let's move on.. I'll skip the next topic (Pools - Where are we? (kiall - recurring)) since we decided to remove that from the recurring agenda.. 17:08:43 if people send me lists of revirews - i will see if i can grt them done this evemning 17:08:50 mugsie: will do.. 17:09:06 #topic Summit Talks Submission (Deadline 2/9) (timsim) 17:09:23 So we need to get our talk submissions in by next monday for Vancouver. 17:09:25 #link https://www.openstack.org/summit/vancouver-2015/call-for-speakers/ 17:09:43 But we haven't really talked about what we want to do all that much, so I figured we should :) 17:10:01 Hah - Yea, do we know who's likely to be able to attend? 17:10:32 if i am presenting, i will be able to attend. :) 17:10:38 Not really (from our point of view). Having an accepted talk would increase those chances. 17:11:43 designate installation workshop, pool manager, designate overview, designate agent - all sound like worthy discussions 17:11:48 Okay - Well, Should we try organize a meet on Friday then to decide and submit something? 17:12:07 Sure. Maybe throw up an etherpad with ideas and rough abstracts before then? 17:12:16 +1 17:12:20 +1 17:12:28 timsim: want to take charge of that? ;) 17:12:37 Kiall: Sure. 17:12:56 #link https://etherpad.openstack.org/p/designate-vancouver-talks 17:13:09 #action timsim to start etherpad and mail everyone with a meet time for Friday :) 17:13:16 and people to fill ideas into ^ etherpad 17:14:07 Cool. We can move on :) 17:14:19 Trying to find the tab with the agenda to do so ;) 17:14:35 #topic Next Sprint 17:14:50 timsim: you added this one I think? 17:14:52 docs 17:14:59 Yep 17:15:00 #topic https://etherpad.openstack.org/p/designate-documentation-sprint 17:15:02 lol 17:15:11 #topic Next Sprint 17:15:16 #link https://etherpad.openstack.org/p/designate-documentation-sprint 17:15:20 Actually just left over from last week, just wanted to remind folks to add their ideas there. 17:15:20 That's better. 17:15:45 Ah, I see ... I've put all mine down, has everyone else? 17:16:04 I haven't taken all the time I should have to have thought about it, but I've added a few. 17:16:17 And - everyone's free (US Morning, EU Afternoon) on Feb 13th to do it? 17:16:38 i'll be there. 17:16:40 Yep 17:16:45 I'' be there 17:16:55 yup 17:16:58 Cool - At the next meet, we should divvy up any areas listed out so people can think about them a little in advance. 17:17:19 But - It looks like only myself and timsim have added suggestions, I've sure we can find more :) 17:17:44 Agreed. 17:18:27 rjrjr_ / mugsie / vinod - anything you think of, please add to the list before next weeks meeting then.. 17:18:35 sure 17:18:54 Okay .. Lets move on so.. 17:18:55 #topic Bug triage (timsim) 17:19:23 Just wondered if we should triage some of the bugs out there, there seem to be quite a few. If not, move on :P 17:19:47 Yea - The "untriaged-bot" list of bugs has been growing, and the list of stuff triaged but not fixed/in-progress is growing too.. 17:20:17 I also think we have a large number of invalid bugs.. 17:20:57 Maybe at some point we get them together and go through them. Or action someone. 17:21:13 Any suggestions on how we can make tame the list more easily? Ideally, not a once off triage meet, something ongoing so we can get the list down and keep it down :) 17:22:23 Should we add a "Bug Triage" item to the agenda, and work through new bugs each week trying to find owners etc? 17:22:33 how about a recurring agenda item for the irc and do it - ideally the last item on the agenda 17:22:36 I think, ideally there would be some designated (lol pun) team of people that looked at and made some determination on new bugs with 2 or 3 days. If they weren't handled during the week we could go through the ones from the week during the meet 17:22:55 s/with 2 or 3/within 2 or 3 17:23:07 I think the people on the meet are probably those same people :) 17:23:23 Granted. 17:23:32 As the project grows it might be a good habit. 17:24:03 I like vinod's idea though. 17:24:05 ++ - But I think for now doing it once a week as a group in the meet works.. 17:24:19 Okay - Anyone disagree? 17:24:30 sounds good to me! 17:24:36 #action kiall to put Weekly Bug Triage on agenda as a recurring item.. 17:24:40 sorted :) 17:24:48 Nice 17:25:11 #topic Open Discussion 17:25:45 over 1/3rd of our outstanding bugs are in "Fix Committed", so that's good. 17:26:11 So - Apologies for being so absent the last little while, myself and mugsie have had 3 days in the office since returning from the christmas break.. We're (almost) done with travel at this point.. And the next one shouldn't be as disruptive as the others. 17:27:10 rjrjr_: yep, it's the In Progress ones I'm worried about :) All the Fix Comitted will be moved to Fix Released tomorrow 17:27:18 once Thierry cut's the k2 release for us 17:27:54 a dozen in "In progress". nice! 17:27:54 Anyone have any off-agenda topics to bring up? 17:28:19 dvorak? 17:28:38 sure 17:29:25 I'm investingating deploying designate for our prod environment, and right now we have a cross-region galera cluster, but rabbit clusters that are local to each region 17:30:06 I'm wondering if it'd be a crazy thing to have local designate instances in each region share the database in order to be able to have domains available in multiple regions 17:30:26 or if that's going to explode in interesting ways 17:30:38 So, I'm assuming you're already OK with the latency introduced with cross-region galera cluster? 17:30:51 yeah, we use it for keystone and for horizon caching 17:31:01 we own the network between the datacenters, so that's managable 17:31:14 horizon session state I mean 17:31:20 What kind of scale are we talking about? 17:31:48 we have separate, local galera clusters for other services that it doesn't make sense to replicate 17:31:56 Okay - Well, In theory if you have all the designate components deployed in each region, sharing a DB, that should work just fine assuming your OK with the synchnous replication lag... EXCEPT for 1 quirk.. 17:32:04 timsim: scale in what dimension? 17:32:29 How many zones/records are you expecting, and how many changes are you expecting to make? 17:33:23 honestly, it's all greenfield from a DNS standpoint right now, so it's hard to say, but I'd say less than low hundreds of tenants, probably relatively static records, other than instance spinup/down, which would probably again be hundreds to low thousands per day 17:33:29 that's all really guessing though 17:33:41 The domain table has a serial # column, and galera does opportunistic locking inside transactions, so 2 changes to one zone at each side can conflict - one will get dupmed out as a SQL Deadlock, which is different behaviour to what would happen with a single non-clustered MySQL.. 17:34:06 Kiall: and in that case will the transaction get retried, or an error returned to the requester? 17:34:32 dvorak: Today - No, I have a review up to make it retry.. but it's got out of date and needs an update: https://review.openstack.org/#/c/134524/ 17:34:38 Once it's merged to master I plan on backporting to Juno 17:35:54 I feel like that can probably work, excluding where changes might be made to a zone in two places at once. Changes might be slow to propogate though, as you'd have two wait for a syncing mechanism to come through and get changes, rather than pushing them out right away (they'll go out right away in one DC, but the other will take longer) 17:36:01 A temporary workaround for that not being merged would be to use a single DB server as "master" for all the regions, the latency is larger again that way, but if the regions are close, it should be OK.. 17:36:30 that'd probably be workable. We already do that inside a given region anyway 17:36:32 (We actually use Galera for HP Cloud DNS, and even within a region only every write to a single cluster member at once..) 17:37:16 timsim: galera is synchronous replication, unlike standard mysql replication.. 17:37:21 I suspect our primary use case for most updates will be instance launch, and if galera takes longer to replicatte than the instance takes to boot, then I guess that'd be very good problem, or a very bad problem 17:37:36 A transaction won't commit until the change has been propagated to all the DB servers in the cluster 17:37:36 for keystone we typically see replication <1-2 seconds 17:37:40 Wait, Kiall with Juno how would you push those changes out to two separate dns deployments? 17:38:07 and I suspect the tokens are a lot bigger transactions than the designate ones would be 17:38:35 dvorak: Well, designate's DB schema has more "hot" points than Keystones.. Specially the domain serial number 17:38:41 Kiall: it has to be certified commitable by the other galera cluster members, but it doesn't have to be commited or visible yet 17:39:02 issuing a new token in Keystone is a simple INSERT, while adding a record in Designate is a INSERT and UPDATE 17:39:23 ie, they have the data, they've validated it won't fail due to conflict, but it isn't necessarily commited on all members when the client returns 17:39:33 Kiall: nod, I'm speaking strictly about the replication lag time 17:39:35 Ah - I didn't know that.. 17:40:13 we have a monitoring metric for cross-region keystone replication, that's how we got into this 17:40:14 Anyway - The other option BTW - Is to deploy all of designate designate in 1 region, and onlt designate-sink in the other region - pointed at the first regions RMQ 17:40:34 we'll get a token in one region, then measure how long it takes to be valid in the other 17:40:46 that's not a bad option either, for initial deployment 17:41:06 ok, so sounds like there are some good options to explore 17:41:07 Yea, I think once retry on deadlock lands and is backported to Juno, the problem mostly goes away 17:41:22 kiall, designate sink will read from one RMQ and write to another? 17:41:24 Kiall: I assume that'd be before kilo releases? 17:41:25 At that points is really all about if your happy to deal with the replication lag 17:41:33 rjrjr_: DOOH 17:41:47 or are you talking about shovelling the traffic between RMQ? 17:41:47 dvorak: that last suggestion won't work, as rjrjr_ found out the hard way a while back. 17:41:54 I don't see the replication lag as an issue 17:42:20 I'm fine with it taking a few seconds for new records to be valid 17:42:56 Yea, for DNS a little bit of lag really isn't the end of the world 17:43:21 fair enough, I'm also ok with *trying* to do cross region DB across if need be also. This is really at the exploratory stages. We may not have a hard requirement for cross-site domains, but I suspect it would be desireable 17:43:38 dvorak: let us know how your testing works out :) 17:43:45 Okay - Any other off topic items? 17:43:51 dvorak, we are in the same boat. rolling out next week! 17:44:11 we are going with regional databases to start, then galera after some testing. 17:44:48 Does Friday at the time of this meeting, or an hour before work for everyone to plan vancouver talks? 17:45:22 timsim: yea, that works for me 17:45:53 +1 17:46:29 which one are we agreeing to. :) 17:46:36 No idea if it'll be accepted, but I optimistically proposed a talk about our deployment that I haven't started :) 17:46:41 rjrjr_: either for me ;) 17:46:47 Cool. I'll schedule it for 16:00 UTC 17:46:48 dvorak: oh nice :) 17:46:56 10 AM texas time. 17:47:11 sounds good. 17:47:18 For two hours, and if we want to go an hour later, we'll do that :P 17:47:51 Alright, I'm all good :) 17:48:10 Okay - Works for me.. :) 17:48:20 Anything else before we call it a day? 17:48:48 Reminder - Add docs bits to the docs sprint etherpad, and code review anything tagged k2 that you can.. tomorrow's the day 17:50:04 Okay - See you guys later, time for me to get on those code reviews :) 17:50:05 #endmeeting