17:00:25 #startmeeting Designate 17:00:26 Meeting started Wed Sep 25 17:00:25 2013 UTC and is due to finish in 60 minutes. The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:31 The meeting name has been set to 'designate' 17:00:51 Heya 17:00:53 Who's about? 17:00:59 here 17:01:00 \o 17:01:06 o/ 17:01:38 Okay :) 17:01:44 #topic Review action items from previous weeks 17:02:02 First up - MySQL BIND9 Docs (CaptTofu) 17:02:12 He's AFK at the moment, but it's still on his todo list :( 17:02:17 BP for Pools/Agent Replacement (mugsie) 17:02:20 mugsie: here? 17:02:23 yup 17:02:29 #link https://wiki.openstack.org/wiki/Designate/Server_Pools 17:02:49 any comments / feedback, let me know 17:03:06 CLI Docs (simonmcc) 17:03:16 and Python Binding Docs (kiall) 17:03:20 yep, up & published to readthedocs 17:03:25 Both are up on RTD 17:03:27 #link http://python-designateclient.readthedocs.org/en/latest/ 17:03:55 Hopefully complete enough to be usable :) 17:04:07 That's great guys, thank you. 17:04:11 indeed! 17:04:22 And .. That's all the open items from previous weeks .. just CaptTofu's to get closed :) 17:04:33 #action CaptTofu MySQL BIND9 Docs 17:04:41 Okay so .. 17:04:44 #topic Talk about standardizing logging 17:04:50 vinodmr: about? 17:05:10 #link https://blueprints.launchpad.net/designate/+spec/standardize-logging 17:05:32 I just wanted to be sure that the logs we use are useful for investigating issues when they arise 17:06:05 So, I'm guessing you guys have done some digging and have areas that could be improved? 17:06:20 Do you know if other openstack projects have a format for logging or standards on what they log to info vs debug? 17:06:56 There was a push in nova maybe 2 releases ago to get INFO+ logging actually useful, they may have some guidelines from that excercise 17:07:21 Before that, running nova without DEBUG logging was basically a requirement. 17:08:13 Do you mean running nova with DEBUG logging? 17:08:36 Yea, about 2 releases ago.. running nova without DEBUG logging enabled basically meant you had no useful logs.. 17:08:41 Somewhat similar to Designate today 17:09:29 So - I reckon they came up with some guidelines on what to log, and at what level.. It would be worth trying to dig those out to help guide us :) 17:09:44 We noticed the logging was not consistent or useful at some parts but did not spend enough time to look through to see what is needed etc 17:10:09 I can take a look at that as an action item 17:10:22 Sure 17:10:40 #action vinodmr Search out any logging guidelines from other OpenStack projects 17:11:37 Anyway, we know the logging is missing in some areas, and set "too low" or inconsistently in others :( 17:11:53 It's certainly something we need to figure out a standard for, and stick to it! 17:11:59 agreed 17:13:02 #action kiall review HP's DEBUG level logs, increase anything generally useful to INFO or above 17:13:31 I'll review our DEBUG logs over the next few days, and try and bump the level on anything that's generally useful for production.. 17:13:40 Probably a good start. 17:13:56 Thanks kiall. That would be helpful 17:14:32 Okay - We'll come back to this next week after I've had a chance to review our logs, and we've had a chance to see what other projects do.. 17:14:59 make sense? :) 17:15:09 Yes it does 17:15:15 sounds perfect 17:15:15 Cool 17:15:16 Okay .. 17:15:21 #topic API v2.0 RecordSets - API Change 17:15:51 So .. while working on the v2 RecordSets API, we've found a issue that will need a change to the API spec. 17:16:30 Essentially, as whole RecordSets are updated in 1 go, we run into some concurrency issues with either multiple users making changes to the same recordset at once 17:16:50 or users and automated services (designate-sink) making changes at once.. 17:18:22 So, I think we need to support both the "old style" and "new style" at once, meaning we would reintroduce /recordsets//records/ to the spec, to allow for single records to be changed independently of each other 17:18:49 I've not had a chance to update the spec yet, but wanted to get everyones thoughts before I do 17:19:05 Thats a good idea 17:19:13 both in the v2 spec? 17:19:33 yeah, its also fits in with the changes I have made for pools / servers, having them as sub resources 17:19:42 So do multiple users have access to the same recordset? 17:20:11 betsy: the spec is not updated yet, but the recordset container will still exist as it allows us to cleanly enforce some details of the RFCs 17:20:12 by multiple user you mean like subuser within an account 17:20:37 Sounds like the concern is that a record set could potentially contain a lot of data, which *might lead to a conflict as more users might need to update it. 17:20:40 zane: well, more concurrency .. 1 user making two API calls at once, or a user and the sink service both modifying a recordset at once etc 17:20:55 kiall: yes 17:21:09 jmcbride: Well, This really hit me as I was updating the designate-sink code.. 17:21:13 ok 17:21:32 In that case, the system itself is modifying customers recordsets, maybe adding a record matching the instance name etc 17:21:52 If that happens while a user is making a call to the same recordset, one would "win" 17:22:10 Is this a potential scenario for all create/update scenarios (its the age old pessimistic/optimistic locking problem) 17:22:27 But would an 'A' record only consist of the IP address or would it be the full record like in v1? 17:22:59 It's most noticeable on updates of recordsets, where the spec as it stands requires you to provide all the records at once.. 17:23:08 e.g. a RRSet with 1.1.1.1 and 2.2.2.2 in it. 17:23:10 To add 3.3.3.3 17:23:38 you POST /domain/ID/recordsets/ID with {"records": [1.1.1.1, 2.2.2.2, 3.3.3.3]} 17:23:54 But.. What if another concurrent API call adds 4.4.4.4, by doing a 17:23:57 you POST /domain/ID/recordsets/ID with {"records": [1.1.1.1, 2.2.2.2, 4.4.4.4]} 17:24:27 (^ pseudocode examples btw) 17:25:00 right. 17:25:18 I just don't see how that's a different problem in RecordSets than exists current in v1 records 17:25:26 *currently 17:25:49 So with this change, you minimize the chance of concurrency issues, but it will still exist - correct? 17:26:09 betsy: so, with the v1 API, you CRUD on 1 record at a time, while with the v2 spec as it is today, you CRUD on the recordset as a whole. 17:26:40 true 17:27:08 I see what you're saying 17:27:13 vinodmr: I think we'll be down to the best case with the change I plan on making to the spec, i.e if two API calls come in to update the IP address of an A record at the exact same time.. One of them really has to "win" 17:27:41 But, if two API calls come in to add/update/delete different records in an RRSet, they will both succeed 17:28:26 But they still both also have to update the RS 17:28:37 The update field in the RS would have to be updated in both cases 17:29:06 So RecordSets would just be full or Record ID's, that can then be changed independently, and unless you're creating or deleting a RecordSet, you need not modify the RecordSet? To create a record you just POST to recordsets/id/records? 17:30:10 tsimmons: Essentially, yes. That said, batch operations on the RRSet should still be in spec (Create a recordset with these 10 records, delete the recordset including all records, replace the records in a RRSet with these 5 etc) 17:30:49 For sure, but if you're just messing with one record, which is most of the time, you don't need to modify the RRSet. 17:31:26 That way you kind of get the best of both worlds, you can do operations with multiple records with RRSets, but you can also do single operations with less concurrency issues. 17:31:42 tsimmons: I think you still have to update the RRSet. The updated date field would need to be changed 17:32:12 tsimmons: yea, I think the "Replace a RRSet's records with these 5" is another case where it's perfectly okay for cases for 2 concurrent calls to have a "winner" 17:32:31 And allowing the records to be managed individually makes the common case easier 17:32:41 With the change you are suggesting, if you get one update to an A record another to delete it - what would the behavior be? 17:32:59 Anyway - I'll have the spec updated before Monday, hopefully you guys will have a chance on monday/tuesday to review before the next week.. 17:33:09 vinodmr: it depends, if the delete is processed first, the update would fail. 17:33:22 and if the update is processed first, the delete should suceed 17:33:37 before the next meet* 17:33:41 Agreed. So the operation to RRSet on a one-record update is just changing the updated_date field, and maybe something else small, but that would be done in one API call. 17:34:23 Pretty much :) 17:34:35 Cool. I like that 17:35:08 #action kiall Update APIv2 Spec (before Monday) with proposed RRset/Record changes 17:35:45 Okay .. Any more Q's before we move onto the last item? 17:36:23 Guess not :) 17:36:26 #topic Pools Functionality 17:36:31 cool 17:36:31 mugsie: you're up .. 17:36:55 if you have had a chance to read the wiki page, you will see it has been broken up into 3 phases 17:37:23 Phase One is nearly done - I need to fix the tests, but the code is in place, and the APIs have been updated 17:37:52 if people want to review the code, the link to the gerrit patch is on that pages... 17:38:28 https://blueprints.launchpad.net/designate/+spec/initial-server-pools 17:38:42 ^ that is the launchpad blueprint for phase one 17:38:52 is there any questions about it? 17:39:23 None now. I will look at the wikipage and the code and might have questions for it next week 17:39:29 I haven't had a chance to review it yet, but I like that you have a link to the gerrit patch and the launchpad blueprint on the wiki page 17:39:51 vinodmr: if you want to ping me whenever its fine 17:40:05 i am in the irc room permanetly 17:40:06 sure 17:40:10 betsy: thanks 17:40:24 I'm hoping to give it a good look as well. 17:40:35 cool 17:40:49 as I am ;) 17:40:59 phase one is *fairly big* 17:41:10 (code additions wise) 17:41:32 +1431, -152 17:41:50 so it probably has a few issues 17:42:46 anything else on it? or questions on the pools stuff in general? 17:43:01 Okay :) So .. It's another one to keep on the list for next week after people (including myself!) have a chance to poke holes and review 17:43:25 Okay ... 17:43:29 #topic Open Discussion 17:43:39 Anything else from anyone? I've got nothing 17:43:45 nothing here 17:44:01 not from me 17:44:04 Maybe just a quick status update on some of the open code reviews? 17:44:28 tsimmons: sure.. 17:45:02 I just wanted to check on one the questions that I had yesterday - about created_by and last_modified_by 17:45:08 Some of the open code reviews should really just be abandoned! 17:45:24 e.g. my mockup of what statsd integration might look like 17:45:47 Yeah, Dirk Mueller hasn't been around either, we might reach out to him if possible and see what's up. 17:46:11 and "WIP Support for Floating IP PTRs" was little more than an experiment, I don't think anyone agreed with that approach ;) 17:46:30 I'm not 100% sure about any others, but I know mine is in a waiting state? Is there anything I can do to help, or is there any update on that? 17:46:35 Re Dirk's, one of them seems invalid (Explicitely convert objects to string) 17:47:02 tsimmons: humm - I thought there was something outstanding in that 17:47:19 Looking again, no. There wasn't. 17:47:26 https://review.openstack.org/#/c/45745/ 17:47:43 ^ I'll review that again this evening, but it should be good to merge :) 17:48:06 Cool, then I'm not sure about eankutse's but I think he's close. 17:48:19 eankutse's is a little harder, he brought up a good point a day or so back 17:48:28 re updating the serial when a server changes 17:48:41 Oh yeahhh. Forgot about that. 17:49:10 What about the powerdns backend - does it handle this? 17:49:24 So .. I'm of 2 minds on that .. Perviousally, we didn't even update the servers at all.. So we're defiantly in a better place even if we don't come up with a fix for the serial issue straight away 17:49:59 But I'm not sure if anyone has given it too much thought on how to fix it yet, I know I've been too busy the last couple of days. 17:51:09 I think it's something we should sit on for a few more days, hopefully come up with a fix before we land the change. 17:52:17 Since the serial # was not updated previously, should that be a separate change to ensure that it is fixed everywhere correctly? 17:53:10 That was my "of 2 minds" bit, we're certainly in a better position with the review as is, but hopefully we can figure out a fix before we merge it :0 17:53:12 :)* 17:54:22 Make sense? 17:54:35 Yup. That's cool. 17:54:58 yes 17:55:07 If someone can't dream up a fix over the next few days, I'd say we push it in and add another bug for that part. 17:55:40 Okay .. Anything else? Nearly hitting on time! 17:55:44 So perhaps an action item to clean up the open reviews? If you think it's necessary. 17:56:10 #action kiall Abandon his useless reviews 17:56:13 I just wanted to check on one the questions that I had yesterday - about created_by and last_modified_by 17:56:16 hahaha 17:56:18 #action kiall Review pending stuff :) 17:56:43 vinodmr: Sure 17:57:08 Currently you mentioned that the modified by and created by are sent as notifications - so this would mean that someone else other than designate needs to track this information 17:57:20 So, we don't currently store that info, and there is precedent in OpenStack projects for storing the user_id who created the "thing".. 17:58:06 with things like concurrent changes in recordsets, if we want to look up who made the change, then it would be helpful to have this information handy 17:58:19 vinodmr: yea, personally I feel that an audit trail is something we shouldn't include in designate, we should provide the hooks necessary to integrate with one, but not implement the whole end -> end audit trail .. 17:58:59 e.g. Deployers with Nova / Quantum and Designate will likely want an audit trail for everything 18:00:05 sure i will take a look at the links and we can revisit this issue later if needed 18:00:39 nothing else from me 18:00:49 Yea, makes sense. I reckon we'll need to add hooks in places to send this info out, but I'm scared of implementing "Audit Trail as a Service" ;) 18:01:15 Okay .. Time, and I think there's another meeting here now .. 18:01:17 Thanks all :) 18:01:22 #endmeeting