17:00:25 <Kiall> #startmeeting Designate
17:00:26 <openstack> Meeting started Wed Sep 25 17:00:25 2013 UTC and is due to finish in 60 minutes.  The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:31 <openstack> The meeting name has been set to 'designate'
17:00:51 <Kiall> Heya
17:00:53 <Kiall> Who's about?
17:00:59 <vinodmr> here
17:01:00 <betsy> \o
17:01:06 <simonmcc> o/
17:01:38 <Kiall> Okay :)
17:01:44 <Kiall> #topic Review action items from previous weeks
17:02:02 <Kiall> First up - MySQL BIND9 Docs (CaptTofu)
17:02:12 <Kiall> He's AFK at the moment, but it's still on his todo list :(
17:02:17 <Kiall> BP for Pools/Agent Replacement (mugsie)
17:02:20 <Kiall> mugsie: here?
17:02:23 <mugsie> yup
17:02:29 <Kiall> #link https://wiki.openstack.org/wiki/Designate/Server_Pools
17:02:49 <mugsie> any comments / feedback, let me know
17:03:06 <Kiall> CLI Docs (simonmcc)
17:03:16 <Kiall> and Python Binding Docs (kiall)
17:03:20 <simonmcc> yep, up & published to readthedocs
17:03:25 <Kiall> Both are up on RTD
17:03:27 <Kiall> #link http://python-designateclient.readthedocs.org/en/latest/
17:03:55 <Kiall> Hopefully complete enough to be usable :)
17:04:07 <tsimmons> That's great guys, thank you.
17:04:11 <simonmcc> indeed!
17:04:22 <Kiall> And .. That's all the open items from previous weeks .. just CaptTofu's to get closed :)
17:04:33 <Kiall> #action CaptTofu MySQL BIND9 Docs
17:04:41 <Kiall> Okay so ..
17:04:44 <Kiall> #topic Talk about standardizing logging
17:04:50 <Kiall> vinodmr: about?
17:05:10 <Kiall> #link https://blueprints.launchpad.net/designate/+spec/standardize-logging
17:05:32 <vinodmr> I just wanted to be sure that the logs we use are useful for investigating issues when they arise
17:06:05 <Kiall> So, I'm guessing you guys have done some digging and have areas that could be improved?
17:06:20 <vinodmr> Do you know if other openstack projects have a format for logging or standards on what they log to info vs debug?
17:06:56 <Kiall> There was a push in nova maybe 2 releases ago to get INFO+ logging actually useful, they may have some guidelines from that excercise
17:07:21 <Kiall> Before that, running nova without DEBUG logging was basically a requirement.
17:08:13 <vinodmr> Do you mean running nova with DEBUG logging?
17:08:36 <Kiall> Yea, about 2 releases ago.. running nova without DEBUG logging enabled basically meant you had no useful logs..
17:08:41 <Kiall> Somewhat similar to Designate today
17:09:29 <Kiall> So - I reckon they came up with some guidelines on what to log, and at what level.. It would be worth trying to dig those out to help guide us :)
17:09:44 <vinodmr> We noticed the logging was not consistent or useful at some parts but did not spend enough time to look through to see what is needed etc
17:10:09 <vinodmr> I can take a look at that as an action item
17:10:22 <Kiall> Sure
17:10:40 <Kiall> #action vinodmr Search out any logging guidelines from other OpenStack projects
17:11:37 <Kiall> Anyway, we know the logging is missing in some areas, and set "too low" or inconsistently in others :(
17:11:53 <Kiall> It's certainly something we need to figure out a standard for, and stick to it!
17:11:59 <mugsie> agreed
17:13:02 <Kiall> #action kiall review HP's DEBUG level logs, increase anything generally useful to INFO or above
17:13:31 <Kiall> I'll review our DEBUG logs over the next few days, and try and bump the level on anything that's generally useful for production..
17:13:40 <Kiall> Probably a good start.
17:13:56 <vinodmr> Thanks kiall.  That would be helpful
17:14:32 <Kiall> Okay - We'll come back to this next week after I've had a chance to review our logs, and we've had a chance to see what other projects do..
17:14:59 <Kiall> make sense? :)
17:15:09 <vinodmr> Yes it does
17:15:15 <betsy> sounds perfect
17:15:15 <Kiall> Cool
17:15:16 <Kiall> Okay ..
17:15:21 <Kiall> #topic API v2.0 RecordSets - API Change
17:15:51 <Kiall> So .. while working on the v2 RecordSets API, we've found a issue that will need a change to the API spec.
17:16:30 <Kiall> Essentially, as whole RecordSets are updated in 1 go, we run into some concurrency issues with either multiple users making changes to the same recordset at once
17:16:50 <Kiall> or users and automated services (designate-sink) making changes at once..
17:18:22 <Kiall> So, I think we need to support both the "old style" and "new style" at once, meaning we would reintroduce /recordsets/<id>/records/<id> to the spec, to allow for single records to be changed independently of each other
17:18:49 <Kiall> I've not had a chance to update the spec yet, but wanted to get everyones thoughts before I do
17:19:05 <zane> Thats a good idea
17:19:13 <betsy> both in the v2 spec?
17:19:33 <mugsie> yeah, its also fits in with the changes I have made for pools / servers, having them as sub resources
17:19:42 <vinodmr> So do multiple users have access to the same recordset?
17:20:11 <Kiall> betsy: the spec is not updated yet, but the recordset container will still exist as it allows us to cleanly enforce some details of the RFCs
17:20:12 <zane> by multiple user you mean like subuser within an account
17:20:37 <jmcbride> Sounds like the concern is that a record set could potentially contain a lot of data, which *might lead to a conflict as more users might need to update it.
17:20:40 <Kiall> zane: well, more concurrency .. 1 user making two API calls at once, or a user and the sink service both modifying a recordset at once etc
17:20:55 <vinodmr> kiall: yes
17:21:09 <Kiall> jmcbride: Well, This really hit me as I was updating the designate-sink code..
17:21:13 <zane> ok
17:21:32 <Kiall> In that case, the system itself is modifying customers recordsets, maybe adding a record matching the instance name etc
17:21:52 <Kiall> If that happens while a user is making a call to the same recordset, one would "win"
17:22:10 <jmcbride> Is this a potential scenario for all create/update scenarios (its the age old pessimistic/optimistic locking problem)
17:22:27 <betsy> But would an 'A' record only consist of the IP address or would it be the full record like in v1?
17:22:59 <Kiall> It's most noticeable on updates of recordsets, where the spec as it stands requires you to provide all the records at once..
17:23:08 <Kiall> e.g. a RRSet with 1.1.1.1 and 2.2.2.2 in it.
17:23:10 <Kiall> To add 3.3.3.3
17:23:38 <Kiall> you POST /domain/ID/recordsets/ID with {"records": [1.1.1.1, 2.2.2.2, 3.3.3.3]}
17:23:54 <Kiall> But.. What if another concurrent API call adds 4.4.4.4, by doing a
17:23:57 <Kiall> you POST /domain/ID/recordsets/ID with {"records": [1.1.1.1, 2.2.2.2, 4.4.4.4]}
17:24:27 <Kiall> (^ pseudocode examples btw)
17:25:00 <betsy> right.
17:25:18 <betsy> I just don't see how that's a different problem in RecordSets than exists current in v1 records
17:25:26 <betsy> *currently
17:25:49 <vinodmr> So with this change, you minimize the chance of concurrency issues, but it will still exist - correct?
17:26:09 <Kiall> betsy: so, with the v1 API, you CRUD on 1 record at a time, while with the v2 spec as it is today, you CRUD on the recordset as a whole.
17:26:40 <betsy> true
17:27:08 <betsy> I see what  you're saying
17:27:13 <Kiall> vinodmr: I think we'll be down to the best case with the change I plan on making to the spec, i.e if two API calls come in to update the IP address of an A record at the exact same time.. One of them really has to "win"
17:27:41 <Kiall> But, if two API calls come in to add/update/delete different records in an RRSet, they will both succeed
17:28:26 <betsy> But they still both also have to update the RS
17:28:37 <betsy> The update field in the RS would have to be updated in both cases
17:29:06 <tsimmons> So RecordSets would just be full or Record ID's, that can then be changed independently, and unless you're creating or deleting a RecordSet, you need not modify the RecordSet? To create a record you just POST to recordsets/id/records?
17:30:10 <Kiall> tsimmons: Essentially, yes. That said, batch operations on the RRSet should still be in spec (Create a recordset with these 10 records, delete the recordset including all records, replace the records in a RRSet with these 5 etc)
17:30:49 <tsimmons> For sure, but if you're just messing with one record, which is most of the time, you don't need to modify the RRSet.
17:31:26 <tsimmons> That way you kind of get the best of both worlds, you can do operations with multiple records with RRSets, but you can also do single operations with less concurrency issues.
17:31:42 <betsy> tsimmons: I think you still have to update the RRSet. The updated date field would need to be changed
17:32:12 <Kiall> tsimmons: yea, I think the "Replace a RRSet's records with these 5" is another case where it's perfectly okay for cases for 2 concurrent calls to have a "winner"
17:32:31 <Kiall> And allowing the records to be managed individually makes the common case easier
17:32:41 <vinodmr> With the change you are suggesting, if you get one update to an A record another to delete it - what would the behavior be?
17:32:59 <Kiall> Anyway - I'll have the spec updated before Monday, hopefully you guys will have a chance on monday/tuesday to review before the next week..
17:33:09 <Kiall> vinodmr: it depends, if the delete is processed first, the update would fail.
17:33:22 <Kiall> and if the update is processed first, the delete should suceed
17:33:37 <Kiall> before the next meet*
17:33:41 <tsimmons> Agreed. So the operation to RRSet on a one-record update is just changing the updated_date field, and maybe something else small, but that would be done in one API call.
17:34:23 <Kiall> Pretty much :)
17:34:35 <tsimmons> Cool. I like that
17:35:08 <Kiall> #action kiall Update APIv2 Spec (before Monday) with proposed RRset/Record changes
17:35:45 <Kiall> Okay .. Any more Q's before we move onto the last item?
17:36:23 <Kiall> Guess not :)
17:36:26 <Kiall> #topic Pools Functionality
17:36:31 <mugsie> cool
17:36:31 <Kiall> mugsie: you're up ..
17:36:55 <mugsie> if you have had a chance to read the wiki page, you will see it has been broken up into 3 phases
17:37:23 <mugsie> Phase One is nearly done - I need to fix the tests, but the code is in place, and the APIs have been updated
17:37:52 <mugsie> if people want to review the code, the link to the gerrit patch is on that pages...
17:38:28 <mugsie> https://blueprints.launchpad.net/designate/+spec/initial-server-pools
17:38:42 <mugsie> ^ that is the launchpad blueprint for phase one
17:38:52 <mugsie> is there any questions about it?
17:39:23 <vinodmr> None now.  I will look at the wikipage and the code and might have questions for it next week
17:39:29 <betsy> I haven't had a chance to review it yet, but I like that you have a link to the gerrit patch and the launchpad blueprint on the wiki page
17:39:51 <mugsie> vinodmr: if you want to ping me whenever its fine
17:40:05 <mugsie> i am in the irc room permanetly
17:40:06 <vinodmr> sure
17:40:10 <mugsie> betsy: thanks
17:40:24 <tsimmons> I'm hoping to give it a good look as well.
17:40:35 <mugsie> cool
17:40:49 <Kiall> as I am ;)
17:40:59 <mugsie> phase one is *fairly big*
17:41:10 <mugsie> (code additions wise)
17:41:32 <mugsie> +1431, -152
17:41:50 <mugsie> so it probably has a few issues
17:42:46 <mugsie> anything else on it? or questions on the pools stuff in general?
17:43:01 <Kiall> Okay :) So .. It's another one to keep on the list for next week after people (including myself!) have a chance to poke holes and review
17:43:25 <Kiall> Okay ...
17:43:29 <Kiall> #topic Open Discussion
17:43:39 <Kiall> Anything else from anyone? I've got nothing
17:43:45 <simonmcc> nothing here
17:44:01 <mugsie> not from me
17:44:04 <tsimmons> Maybe just a quick status update on some of the open code reviews?
17:44:28 <Kiall> tsimmons: sure..
17:45:02 <vinodmr> I just wanted to check on one the questions that I had yesterday - about created_by and last_modified_by
17:45:08 <Kiall> Some of the open code reviews should really just be abandoned!
17:45:24 <Kiall> e.g. my mockup of what statsd integration might look like
17:45:47 <tsimmons> Yeah, Dirk Mueller hasn't been around either, we might reach out to him if possible and see what's up.
17:46:11 <Kiall> and "WIP Support for Floating IP PTRs" was little more than an experiment, I don't think anyone agreed with that approach ;)
17:46:30 <tsimmons> I'm not 100% sure about any others, but I know mine is in a waiting state? Is there anything I can do to help, or is there any update on that?
17:46:35 <Kiall> Re Dirk's, one of them seems invalid (Explicitely convert objects to string)
17:47:02 <Kiall> tsimmons: humm - I thought there was something outstanding in that
17:47:19 <Kiall> Looking again, no. There wasn't.
17:47:26 <Kiall> https://review.openstack.org/#/c/45745/
17:47:43 <Kiall> ^ I'll review that again this evening, but it should be good to merge :)
17:48:06 <tsimmons> Cool, then I'm not sure about eankutse's but I think he's close.
17:48:19 <Kiall> eankutse's is a little harder, he brought up a good point a day or so back
17:48:28 <Kiall> re updating the serial when a server changes
17:48:41 <tsimmons> Oh yeahhh. Forgot about that.
17:49:10 <vinodmr> What about the powerdns backend - does it handle this?
17:49:24 <Kiall> So .. I'm of 2 minds on that .. Perviousally, we didn't even update the servers at all.. So we're defiantly in a better place even if we don't come up with a fix for the serial issue straight away
17:49:59 <Kiall> But I'm not sure if anyone has given it too much thought on how to fix it yet, I know I've been too busy the last couple of days.
17:51:09 <Kiall> I think it's something we should sit on for a few more days, hopefully come up with a fix before we land the change.
17:52:17 <vinodmr> Since the serial # was not updated previously, should that be a separate change to ensure that it is fixed everywhere correctly?
17:53:10 <Kiall> That was my "of 2 minds" bit, we're certainly in a better position with the review as is, but hopefully we can figure out a fix before we merge it :0
17:53:12 <Kiall> :)*
17:54:22 <Kiall> Make sense?
17:54:35 <tsimmons> Yup. That's cool.
17:54:58 <vinodmr> yes
17:55:07 <Kiall> If someone can't dream up a fix over the next few days, I'd say we push it in and add another bug for that part.
17:55:40 <Kiall> Okay .. Anything else? Nearly hitting on time!
17:55:44 <tsimmons> So perhaps an action item to clean up the open reviews? If you think it's necessary.
17:56:10 <Kiall> #action kiall Abandon his useless reviews
17:56:13 <vinodmr> I just wanted to check on one the questions that I had yesterday - about created_by and last_modified_by
17:56:16 <tsimmons> hahaha
17:56:18 <Kiall> #action kiall Review pending stuff :)
17:56:43 <Kiall> vinodmr: Sure
17:57:08 <vinodmr> Currently you mentioned that the modified by and created by are sent as notifications - so this would mean that someone else other than designate needs to track this information
17:57:20 <Kiall> So, we don't currently store that info, and there is precedent in OpenStack projects for storing the user_id who created the "thing"..
17:58:06 <vinodmr> with things like concurrent changes in recordsets, if we want to look up who made the change, then it would be helpful to have this information handy
17:58:19 <Kiall> vinodmr: yea, personally I feel that an audit trail is something we shouldn't include in designate, we should provide the hooks necessary to integrate with one, but not implement the whole end -> end audit trail ..
17:58:59 <Kiall> e.g. Deployers with Nova / Quantum and Designate will likely want an audit trail for everything
18:00:05 <vinodmr> sure i will take a look at the links and we can revisit this issue later if needed
18:00:39 <vinodmr> nothing else from me
18:00:49 <Kiall> Yea, makes sense. I reckon we'll need to add hooks in places to send this info out, but I'm scared of implementing "Audit Trail as a Service" ;)
18:01:15 <Kiall> Okay .. Time, and I think there's another meeting here now ..
18:01:17 <Kiall> Thanks all :)
18:01:22 <Kiall> #endmeeting