| *** boris-42 has joined #openstack-dns | 00:04 | |
| *** ducttape_ has joined #openstack-dns | 00:10 | |
| *** rudrajit has joined #openstack-dns | 00:16 | |
| *** ducttape_ has quit IRC | 00:23 | |
| *** km has quit IRC | 00:52 | |
| *** km__ has joined #openstack-dns | 00:52 | |
| *** km__ is now known as Guest71872 | 00:53 | |
| *** ducttape_ has joined #openstack-dns | 01:04 | |
| *** stanzgy has joined #openstack-dns | 01:12 | |
| *** bpokorny has joined #openstack-dns | 01:19 | |
| *** ccneill has joined #openstack-dns | 01:28 | |
| *** EricGonczer_ has joined #openstack-dns | 01:34 | |
| *** bpokorny has quit IRC | 01:53 | |
| *** ducttape_ has quit IRC | 01:56 | |
| *** ducttape_ has joined #openstack-dns | 01:57 | |
| *** ccneill has quit IRC | 01:59 | |
| *** ducttape_ has quit IRC | 02:05 | |
| *** EricGonczer_ has quit IRC | 02:22 | |
| *** rudrajit has quit IRC | 02:32 | |
| *** rudrajit has joined #openstack-dns | 02:39 | |
| *** ducttape_ has joined #openstack-dns | 02:45 | |
| *** EricGonczer_ has joined #openstack-dns | 02:47 | |
| *** ducttape_ has quit IRC | 02:59 | |
| *** EricGonczer_ has quit IRC | 03:02 | |
| *** Guest71872 has quit IRC | 03:11 | |
| *** km has joined #openstack-dns | 03:11 | |
| *** ducttape_ has joined #openstack-dns | 03:14 | |
| *** stanzgy has quit IRC | 03:24 | |
| *** stanzgy has joined #openstack-dns | 03:36 | |
| *** EricGonczer_ has joined #openstack-dns | 03:41 | |
| *** EricGonczer_ has quit IRC | 03:43 | |
| *** ducttape_ has quit IRC | 03:45 | |
| *** rektide has quit IRC | 04:15 | |
| *** bpokorny has joined #openstack-dns | 04:22 | |
| *** bpokorny has quit IRC | 04:30 | |
| *** ccneill has joined #openstack-dns | 05:55 | |
| *** ccneill has quit IRC | 06:00 | |
| *** jasonsb has joined #openstack-dns | 06:52 | |
| *** chlong_ has quit IRC | 07:30 | |
| *** jasonsb has quit IRC | 07:33 | |
| *** jasonsb has joined #openstack-dns | 07:36 | |
| *** wolsen has quit IRC | 07:57 | |
| *** wolsen has joined #openstack-dns | 08:00 | |
| *** pcaruana has joined #openstack-dns | 08:05 | |
| *** nyechiel_ has joined #openstack-dns | 08:44 | |
| *** jschwarz has joined #openstack-dns | 08:53 | |
| *** nyechiel_ has quit IRC | 08:53 | |
| *** rudrajit has quit IRC | 09:01 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add support for getting Service Status https://review.openstack.org/280773 | 09:13 |
|---|---|---|
| *** jordanP has joined #openstack-dns | 09:23 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add support for getting Service Status https://review.openstack.org/280773 | 09:33 |
| *** chlong_ has joined #openstack-dns | 10:06 | |
| *** kei_yama has quit IRC | 10:41 | |
| *** nyechiel_ has joined #openstack-dns | 10:43 | |
| *** nyechiel_ has quit IRC | 10:55 | |
| *** nyechiel_ has joined #openstack-dns | 10:56 | |
| *** nyechiel_ has quit IRC | 11:05 | |
| *** stanzgy has quit IRC | 11:15 | |
| *** jet-rongl has joined #openstack-dns | 11:30 | |
| *** jet-rongl has quit IRC | 11:33 | |
| *** jet-rongl has joined #openstack-dns | 11:34 | |
| *** jet-rongl has quit IRC | 11:36 | |
| *** andrewbogott has quit IRC | 11:39 | |
| *** ryanpetrello has quit IRC | 11:40 | |
| *** rackertom has quit IRC | 11:40 | |
| *** fyxim has quit IRC | 11:40 | |
| *** zigo has quit IRC | 11:40 | |
| *** bauruine has quit IRC | 11:40 | |
| *** d34dh0r53 has quit IRC | 11:40 | |
| *** dhellmann has quit IRC | 11:41 | |
| *** zigo has joined #openstack-dns | 11:41 | |
| *** andrewbogott has joined #openstack-dns | 11:43 | |
| *** fyxim has joined #openstack-dns | 11:43 | |
| *** rackertom has joined #openstack-dns | 11:43 | |
| *** jet-rongl has joined #openstack-dns | 11:44 | |
| *** ryanpetrello has joined #openstack-dns | 11:46 | |
| *** dhellmann has joined #openstack-dns | 11:46 | |
| *** d34dh0r53 has joined #openstack-dns | 11:46 | |
| *** bauruine has joined #openstack-dns | 11:46 | |
| *** jet-rongl has quit IRC | 11:46 | |
| *** jet-rongl has joined #openstack-dns | 11:49 | |
| *** km has quit IRC | 11:59 | |
| *** chlong_ has quit IRC | 12:17 | |
| *** jet-rongl has quit IRC | 12:29 | |
| *** nyechiel_ has joined #openstack-dns | 12:47 | |
| *** ducttape_ has joined #openstack-dns | 13:12 | |
| *** ducttape_ has quit IRC | 13:24 | |
| *** andrewbogott has quit IRC | 13:55 | |
| *** andrewbogott has joined #openstack-dns | 13:55 | |
| *** rsyed_away is now known as rsyed | 13:59 | |
| mugsie | timsim: whan you wake up - https://review.openstack.org/251853 | 14:01 |
| *** richm has joined #openstack-dns | 14:02 | |
| *** testing-tester has joined #openstack-dns | 14:03 | |
| *** testing-tester has left #openstack-dns | 14:03 | |
| *** chlong_ has joined #openstack-dns | 14:06 | |
| *** openstackgerrit has quit IRC | 14:17 | |
| *** openstackgerrit has joined #openstack-dns | 14:17 | |
| *** ChanServ sets mode: +v openstackgerrit | 14:17 | |
| *** jet-rongl has joined #openstack-dns | 14:23 | |
| elarson | federico3: out of curiosity, any reason for SomethingTest vs TestSomething in test class names/ | 14:23 |
| elarson | names?* | 14:23 |
| federico3 | elarson: the majority of test classes are named SomethingTest but we are not b | 14:26 |
| federico3 | -eing consitent | 14:26 |
| federico3 | 71 SomethingTest VS 24 TestSomething :( | 14:26 |
| federico3 | tools like nose would expect a specific format instead | 14:27 |
| elarson | pytest usually picks up TestSomething or unittest based tests, so I was just curious your feelings | 14:28 |
| * elarson prefers the TestSomething, but mostly b/c it is familiar | 14:28 | |
| federico3 | anything is better than random | 14:29 |
| federico3 | TestSomething is also consistent with "def test_foo()" | 14:30 |
| openstackgerrit | Merged openstack/designate: Add delayed NOTIFY https://review.openstack.org/247588 | 14:42 |
| openstackgerrit | Merged openstack/python-designateclient: Updated from global requirements https://review.openstack.org/282770 | 14:42 |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Randomize SOA refresh interval https://review.openstack.org/279475 | 14:54 |
| *** ducttape_ has joined #openstack-dns | 15:00 | |
| *** ducttape_ has quit IRC | 15:00 | |
| *** ducttape_ has joined #openstack-dns | 15:00 | |
| openstackgerrit | Graham Hayes proposed openstack/designate: WIP: Add scheduler for pools https://review.openstack.org/278536 | 15:04 |
| timsim | Ugh mugsie Kiall this change looks like it times out on a test very very rarely: https://review.openstack.org/#/c/272316/ we probably shouldn't merge if it's going to do what it did in the gate to random changes now and then. | 15:21 |
| timsim | Well...actually, that's the periodic one. | 15:22 |
| timsim | That doesn't make any sense. | 15:22 |
| *** jet-rongl has quit IRC | 15:22 | |
| timsim | I had assumed that'd be the poll-deletes one because that actually adds work. | 15:22 |
| timsim | That's probably just regular entropy. | 15:22 |
| *** jet-rongl has joined #openstack-dns | 15:22 | |
| Kiall | Ah, ekarlso's got a similar fail on one of his patches. | 15:22 |
| Kiall | I think we have another $random gate failure merged | 15:22 |
| timsim | neat | 15:23 |
| *** pglass has joined #openstack-dns | 15:26 | |
| Kiall | timsim: yea, we have a gate issue :( https://review.openstack.org/#/c/282469/ | 15:27 |
| timsim | http://replygif.net/i/127 | 15:28 |
| *** jordanP has quit IRC | 15:33 | |
| elarson | the gate... https://media.giphy.com/media/SwAf9VVJRMbok/giphy.gif | 15:36 |
| openstackgerrit | Merged openstack/designate: Fix V1 Quotas API Extension https://review.openstack.org/251853 | 15:38 |
| *** nkinder has joined #openstack-dns | 15:50 | |
| *** penick has joined #openstack-dns | 15:53 | |
| *** nyechiel_ has quit IRC | 15:56 | |
| *** nyechiel_ has joined #openstack-dns | 15:56 | |
| *** jet-rongl has quit IRC | 15:58 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add AttributeListObjectMixin class https://review.openstack.org/280647 | 16:02 |
| Kiall | mugsie / timsim / ekarlso / elarson.. that gate fail, has anyone seen it affect the powerDNS gate yet? | 16:02 |
| *** penick has quit IRC | 16:04 | |
| Kiall | Nasty URL incoming | 16:04 |
| Kiall | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22delete%20failed%20%5C%5C(client%20error%5C%5C)%5C%5C:%20Unable%20to%20complete%20operation%20on%20subnet%5C%22%20AND%20message:%5C%22One%20or%20more%20ports%20have%20an%20IP%20allocation%20from%20this%20subnet%5C%22%20AND%20tags:%5C%22screen-q-svc.txt%5C%22%20AND%20voting:1&from=864000s | 16:04 |
| Kiall | first occourance was 2016-02-15T14:42:44.563+00:00 | 16:05 |
| Kiall | affected bind9 50 times, powerdns 2 times | 16:05 |
| *** EricGonczer_ has joined #openstack-dns | 16:07 | |
| *** EricGonczer_ has quit IRC | 16:11 | |
| elarson | So, writing a test for https://bugs.launchpad.net/designate/+bug/1548331 I'd like to move this out of a decorator: https://github.com/openstack/designate/blob/master/designate/central/service.py#L121 | 16:22 |
| openstack | Launchpad bug 1548331 in Designate "The zone records quota are not enforced" [Undecided,New] - Assigned to Eric Larson (eric-larson) | 16:22 |
| elarson | the reason being is that unwrapping that decorator is really rough in order to test it w/o the db | 16:22 |
| * elarson would prefer to use a context manager explicitly in with the caller | 16:23 | |
| elarson | hmm... maybe I can salvage it... | 16:24 |
| elarson | it is used a lot | 16:24 |
| Kiall | elarson: I'm not convinced that was ever the right place for it! | 16:24 |
| Kiall | (bpth physical location of code, and placement of the TX wrapper | 16:24 |
| elarson | ok, I'll try moving it around and seeing what folks think | 16:25 |
| *** pcaruana has quit IRC | 16:27 | |
| *** ccneill has joined #openstack-dns | 16:28 | |
| openstackgerrit | Merged openstack/designate: Updated from global requirements https://review.openstack.org/282469 | 16:31 |
| *** wolsen has quit IRC | 16:34 | |
| *** wolsen has joined #openstack-dns | 16:42 | |
| *** bpokorny has joined #openstack-dns | 16:46 | |
| openstackgerrit | Chaozhe Chen(ccz) proposed openstack/designate: Stop using WritableLogger() which is deprecated. https://review.openstack.org/283165 | 16:47 |
| *** james_li has joined #openstack-dns | 16:49 | |
| *** bpokorny has quit IRC | 16:51 | |
| *** bpokorny has joined #openstack-dns | 16:51 | |
| *** jschwarz has quit IRC | 16:56 | |
| *** ccneill has quit IRC | 17:09 | |
| *** ccneill has joined #openstack-dns | 17:15 | |
| *** ccneill has quit IRC | 17:17 | |
| *** fawadkhaliq has joined #openstack-dns | 17:18 | |
| *** ccneill has joined #openstack-dns | 17:18 | |
| *** fawadkhaliq has quit IRC | 17:19 | |
| *** penick has joined #openstack-dns | 17:28 | |
| *** jasonsb has quit IRC | 17:30 | |
| *** fawadkhaliq has joined #openstack-dns | 17:42 | |
| *** fawadkhaliq has quit IRC | 17:51 | |
| openstackgerrit | Merged openstack/designate: Update periodic-sync-seconds help https://review.openstack.org/270332 | 17:57 |
| *** ducttape_ has quit IRC | 17:57 | |
| *** eandersson has joined #openstack-dns | 18:01 | |
| eandersson | Hey | 18:01 |
| eandersson | Anyone with some experience doing a live upgrade from Designate Kilo to Liberty | 18:02 |
| *** ducttape_ has joined #openstack-dns | 18:08 | |
| *** rudrajit has joined #openstack-dns | 18:08 | |
| openstackgerrit | Merged openstack/designate: Improve performance and utility of Recovery https://review.openstack.org/272316 | 18:11 |
| openstackgerrit | Merged openstack/designate: Actually poll for zone deletes https://review.openstack.org/274291 | 18:11 |
| *** rudrajit has quit IRC | 18:13 | |
| *** rudrajit has joined #openstack-dns | 18:24 | |
| eandersson | First issue is that all my requests are getting unauthorized :( | 18:24 |
| mugsie | eandersson: really? is there any stack trace in the -api logs? | 18:26 |
| eandersson | I had this with the latest version of Kilo as well. | 18:26 |
| eandersson | It's not working with domain-tokens. | 18:26 |
| eandersson | but might just be a policy or something I forgot to update | 18:27 |
| mugsie | ah. I remember somehting about this ... your talking about domain scoped keystone tokens? | 18:27 |
| eandersson | Yea | 18:27 |
| *** rsyed is now known as rsyed_away | 18:27 | |
| mugsie | as far as I know we need a project scoped token - as we use the project_id for a ton | 18:28 |
| eandersson | It's odd though as I do see some requests successful from our monitoring tools | 18:28 |
| mugsie | we had a bug when with a domain scoped token we would just store the zone / recordset in the db with no project id | 18:28 |
| mugsie | is there any reason you are using domain scoped tokens? | 18:29 |
| eandersson | It's the design of our cloud | 18:30 |
| eandersson | :D | 18:30 |
| eandersson | I can't remember exactly why we went down that route though. | 18:30 |
| mugsie | is there any reason you are using domain scoped tokens? | 18:34 |
| mugsie | gha | 18:34 |
| mugsie | hit up by accident | 18:34 |
| eandersson | heh | 18:34 |
| eandersson | Not really seeing an error message in the logs. | 18:34 |
| eandersson | > GET /v1/servers HTTP/1.1 401 | 18:34 |
| eandersson | With debug logs I see it return a token with credentials etc. | 18:35 |
| eandersson | Roles include admin | 18:35 |
| eandersson | roles [{"id":"....", "name "admin"} | 18:36 |
| *** rektide has joined #openstack-dns | 18:36 | |
| mugsie | ah | 18:36 |
| mugsie | https://github.com/openstack/designate/commit/ae235cba3c5bda5d1da6816135400107660a2794 | 18:36 |
| eandersson | That is the patch that broke Kilo | 18:37 |
| mugsie | https://bugs.launchpad.net/designate/+bug/1460187 | 18:37 |
| openstack | Launchpad bug 1460187 in Designate kilo "Designate GET and POST /v1/domains has auth issue" [Critical,Fix committed] - Assigned to Kiall Mac Innes (kiall) | 18:37 |
| eandersson | I mentioned it in IRC here a few months ago | 18:37 |
| eandersson | I had to manually patch it in Kilo when I upgraded. | 18:37 |
| ekarlso | eandersson: hey man :P | 18:38 |
| mugsie | I must have missed it. It seems that we do not know what to do with domain scoped tokens still. Seen as domains are constantly changing (there is going to projects that are now domains?? ) I am not sure what the correct route it | 18:38 |
| mugsie | route is * | 18:38 |
| eandersson | hey ekarlso !! <3 | 18:39 |
| mugsie | wiht your patch, all users will see all the domains though, right? | 18:39 |
| eandersson | at least that isn't how it works for us | 18:40 |
| eandersson | it's per tenant afaik | 18:40 |
| eandersson | I still need to use --all-tenants as an admin to see all domains | 18:40 |
| *** rudrajit_ has joined #openstack-dns | 18:40 | |
| eandersson | At least I know that it isn't a config issue | 18:41 |
| *** rudrajit has quit IRC | 18:42 | |
| eandersson | I'll just apply that patch again. | 18:43 |
| mugsie | ok. i would be interested in seeing how domain scoped tokens are working with tenant level multi tenancy, but that may be a problem for another day | 18:45 |
| eandersson | hmm actually reverting the patch does not seem to work in Liberty :( | 18:46 |
| eandersson | oh nvm I commented out the wrong code lol | 18:46 |
| eandersson | success | 18:46 |
| mugsie | :) | 18:47 |
| eandersson | How is openstackclient support coming along? | 18:48 |
| eandersson | :D | 18:48 |
| eandersson | Is that something being worked on? | 18:48 |
| ekarlso | eandersson: eh, openstackclient u mean the cli ? | 18:49 |
| eandersson | Yep | 18:50 |
| ekarlso | eandersson: been in for a long time :P | 18:51 |
| ekarlso | or relatively long in terms of v2 support :p | 18:51 |
| eandersson | oh | 18:51 |
| mugsie | there is a few rough edges, but it is all thee | 18:51 |
| mugsie | there* | 18:51 |
| eandersson | nice | 18:51 |
| eandersson | interesting > ERROR: list index out of range | 18:53 |
| eandersson | when trying to delete a record in Liberty | 18:53 |
| *** rsyed_away is now known as rsyed | 18:56 | |
| mugsie | is there a full trace? I think I have seen that before .. | 19:01 |
| eandersson | http://paste.openstack.org/show/487796/ | 19:02 |
| eandersson | Upgrade to designateclient 2.0.0, but was the same in 1.5.0 | 19:03 |
| mugsie | looks like a bug - ekarlso ^^ | 19:03 |
| eandersson | Not sure what is happening, because the request comes back as 200, but it then errors out and nothing happens | 19:06 |
| eandersson | the record is still there | 19:06 |
| eandersson | > DEBUG: "DELETE /v1/domains/<xxx>/records/<yyyy> HTTP/1.1" 200 0 | 19:08 |
| eandersson | > DEBUG: RESP: [200] Date: Mon, 22 Feb 2016 19:01:49 GMT Connection: keep-alive Content-Type: text/html; charset=utf-8 Content-Length: 0 X-Openstack-Request-Id: req-f86880e9-62e2-4275-8309-f6b509ee5761 | 19:08 |
| eandersson | Oh it's getting a 404 in designate-api | 19:09 |
| eandersson | didnt see that at first | 19:09 |
| mugsie | the record is probably still in a DELETING state ... so it will show for a while, until the pool-manager sees that it is gone | 19:09 |
| eandersson | I even restarted the pool-manager | 19:10 |
| eandersson | just in case it's that deadlock bug | 19:10 |
| eandersson | trying a full designate restart | 19:11 |
| mugsie | if you do an `openstack recordset list domain.tld.` what does it show? | 19:11 |
| *** ccneill has quit IRC | 19:13 | |
| eandersson | checked the db and it's stuck pending | 19:14 |
| openstackgerrit | Graham Hayes proposed openstack/designate: WIP: Add scheduler for pools https://review.openstack.org/278536 | 19:15 |
| openstackgerrit | Graham Hayes proposed openstack/designate: Move zone masters to a new table https://review.openstack.org/283216 | 19:15 |
| *** james_li has quit IRC | 19:16 | |
| *** rudrajit has joined #openstack-dns | 19:19 | |
| openstackgerrit | Graham Hayes proposed openstack/designate: Move zone masters to a new table https://review.openstack.org/283216 | 19:21 |
| openstackgerrit | Graham Hayes proposed openstack/designate: Add AttributeListObjectMixin class https://review.openstack.org/280647 | 19:21 |
| openstackgerrit | Graham Hayes proposed openstack/designate: WIP: Add scheduler for pools https://review.openstack.org/278536 | 19:21 |
| mugsie | eandersson: crap. I am just running out of the office to catch a bus - will be back in an hour or so. | 19:22 |
| eandersson | thanks mugsie | 19:22 |
| *** rudrajit_ has quit IRC | 19:22 | |
| eandersson | think I have to head home, but I'll be back tomorrow =] | 19:23 |
| *** rudrajit_ has joined #openstack-dns | 19:27 | |
| *** ducttape_ has quit IRC | 19:29 | |
| *** rudrajit has quit IRC | 19:30 | |
| *** ccneill has joined #openstack-dns | 19:34 | |
| *** rsyed is now known as rsyed_away | 19:44 | |
| andrewbogott | federico3: I’m back looking at https://github.com/openstack/designate/blob/stable/kilo/doc/source/backends/powerdns.rst — some of those settings go in subsections, right? e.g. doesn’t the ‘connection’ setting go under '[storage:sqlalchemy]’? | 19:44 |
| *** ducttape_ has joined #openstack-dns | 19:44 | |
| *** james_li has joined #openstack-dns | 19:46 | |
| andrewbogott | heh, the more I read that doc page the less sense it makes :( | 19:48 |
| andrewbogott | There’s sqlite:///$pystatepath/powerdns.sqlite, no explanation of what that db is or where it comes from | 19:50 |
| andrewbogott | and then later connection = mysql://<username>:<password>@<host>:<port>/<dbname> which suggests that designate-central is writing directly to the pdns database, even though everyone agrees that shouldn’t happen | 19:50 |
| andrewbogott | so, I’m back to square one. Could use advice from anyone who has made this work in kilo. | 19:51 |
| *** penick has quit IRC | 19:53 | |
| *** james_li has quit IRC | 19:56 | |
| *** james_li has joined #openstack-dns | 19:56 | |
| *** mlavalle has joined #openstack-dns | 19:57 | |
| pglass | andrewbogott: i don't run powerdns, but powerdns stores all of it's zones in a database. does powerdns have a method to manipulate zones without directly touching the pdns database? | 19:57 |
| *** penick has joined #openstack-dns | 20:05 | |
| *** rsyed_away is now known as rsyed | 20:11 | |
| andrewbogott | pglass: I’m told that as of kilo, everything should be done with mdns and xfr | 20:19 |
| andrewbogott | ah, you mean that you think we do db writes for zones but xfr for records? | 20:19 |
| timsim | I believe it does db writes to create zones, and XFRs to populate the records, yeah. | 20:20 |
| *** mugsie has quit IRC | 20:20 | |
| *** partner has quit IRC | 20:20 | |
| *** ekarlso has quit IRC | 20:20 | |
| *** lkoranda has quit IRC | 20:20 | |
| *** partner has joined #openstack-dns | 20:20 | |
| *** mugsie has joined #openstack-dns | 20:20 | |
| *** ekarlso has joined #openstack-dns | 20:21 | |
| andrewbogott | what, then, was the advantage of ripping out the code that just did everything via db rights? | 20:21 |
| andrewbogott | “This is too easy to configure: | 20:21 |
| andrewbogott | “ ? | 20:21 |
| *** lkoranda has joined #openstack-dns | 20:21 | |
| timsim | I could be wrong. I probably am | 20:21 |
| federico3 | andrewbogott: most likely sqlite was just an example, used for testing, maybe, but in any case you want to use mysql | 20:22 |
| andrewbogott | timsim: and that’s the end state? Or some kind of transition from ‘all db’ to ‘all xfr’? | 20:24 |
| andrewbogott | ugh, sorry netsplit | 20:25 |
| Kiall | andrewbogott: there's no standard for zone creates / deletes, so we still do that per backend type.. For zone updates, we use zone transfers now. It eliminates a whole pile of race conditions we had, and makes working with the likes of BIND much easier as we're not trying to render a file on disk to N servers for every change to a zone. | 20:27 |
| *** nyechiel_ has quit IRC | 20:27 | |
| andrewbogott | timsim: the one install I have is configured for both xfr and db writes. So you are probably right about how it works, that’s just ridiculous | 20:28 |
| timsim | It's really not | 20:28 |
| andrewbogott | Kiall: ok, so — both. That’s unfortunate but I can live with it. | 20:30 |
| Kiall | it kinda is... a standard for zone creates updates would make a pile more sense.. but it doesn't exist :( | 20:30 |
| Kiall | ehh | 20:30 |
| Kiall | a standard for creates a deletes* | 20:30 |
| Kiall | and* | 20:30 |
| * Kiall do can't today english | 20:31 | |
| andrewbogott | so if I want to have two resolvers... | 20:31 |
| timsim | Just write that RFC Kiall | 20:31 |
| andrewbogott | I really can only have one of them talk directly to designate | 20:31 |
| andrewbogott | (in Juno I had both resolvers backed with a single db, which worked since the resolvers had a read-only relationship with the db, the only writer was designate) | 20:31 |
| Kiall | andrewbogott: you can, ypou setup 1 pool target (the DB with type=powerdns), and 2x pool nameservers (with the DNS details) | 20:32 |
| Kiall | oh - missed the read only bit.. No, powerdns will write the zone contents to the DB | 20:32 |
| andrewbogott | Kiall: sorry, still trying to understand the 1 pool target vs 2x pool nameservers thing | 20:33 |
| andrewbogott | doesn’t that still have two nameservers competing for writes to a single db? | 20:33 |
| Kiall | Yes and no, PowerDNS will obtain a write lock on the DB from whichever gets there first is the one to write the contents | 20:34 |
| andrewbogott | so far, we are describing my current setup, which falls over quite a bit | 20:35 |
| *** jmcbride has joined #openstack-dns | 20:35 | |
| *** jmcbride has quit IRC | 20:35 | |
| Kiall | bah - 3rd disconnect from IRC. What's going on with freenode -_- | 20:35 |
| andrewbogott | yeah, I keep getting dropped too | 20:35 |
| *** jmcbride has joined #openstack-dns | 20:35 | |
| Kiall | So, I missed the beginning of the convo.. what does falling over mean? | 20:36 |
| andrewbogott | You didn’t miss it, it’s just why I’m here. | 20:36 |
| andrewbogott | I don’t have a very complete theory. Periodically my resolvers just seize up for a few minutes and have to be restarted. | 20:37 |
| andrewbogott | When I look at the logs, of course they are crazy, a zillion xfr requests flying left and right. | 20:37 |
| Kiall | PowerDNS itself needs restarting? | 20:37 |
| andrewbogott | yeah | 20:37 |
| *** krotscheck has quit IRC | 20:37 | |
| andrewbogott | designate seems fine through all this | 20:37 |
| andrewbogott | So, working theory is: having two pdns instances sharing a database worked with Juno when they never wrote to the db. But adding axfr on top of that produced a world of races. | 20:38 |
| Kiall | So, is it lots of small zones? a few big zones? or a few small zones? | 20:38 |
| andrewbogott | Three zones, they haven’t changed a single time since initial setup | 20:38 |
| Kiall | As in, not even a record in the zones changed? | 20:39 |
| andrewbogott | No, I mean I’m not adding or removing domains at all | 20:39 |
| andrewbogott | I am adding and removing records all the time | 20:39 |
| andrewbogott | generally around 800 records per zone at any one time | 20:39 |
| andrewbogott | (sink is the only thing I’m using designate for at the moment) | 20:40 |
| Kiall | Okay, so nothing large.... | 20:40 |
| andrewbogott | (hoping to add a horizon config for floating IPs and domains once this is resolved) | 20:40 |
| Kiall | Do all the pDNS servers lock up at once? or just 1 at a time? | 20:40 |
| andrewbogott | That’s a good question, and I’m not sure. I think it’s only one of them. | 20:41 |
| andrewbogott | I have a primary and a failover, the failover gets essentially 0 traffic | 20:41 |
| andrewbogott | so even if it did lock we wouldn’t notice | 20:41 |
| andrewbogott | Kiall: bear in mind that just because my config is sort of working, it is almost certainly not ‘correct' | 20:42 |
| andrewbogott | Since the docs (e.g. https://github.com/openstack/designate/blob/stable/kilo/doc/source/backends/powerdns.rst) are pretty much incoherent about config | 20:42 |
| andrewbogott | it’s trial and error for the most part | 20:42 |
| andrewbogott | I can c/p my designate config if that’s of interest. | 20:42 |
| andrewbogott | I’ve also noticed that despite those docs saying "domain_type NATIVE,” the domains that designate actually creates are set to ‘SLAVE,’ and ‘SLAVE’ is a string literal in the code. | 20:44 |
| andrewbogott | (no idea if that’s related or just a random grievance) | 20:44 |
| Kiall | 5th disconnect WTF -_- At least it's keeping the history! | 20:44 |
| Kiall | Yea, the way we interact with DNS servers has changed LOTS and the docs really didn't keep up - we messed up there :( | 20:44 |
| Kiall | Slowly getting things back in shape | 20:44 |
| Kiall | 6th. Okay, next time you see a lockup, as it's hard to tell what's happening from just this info.. Can you: | 20:45 |
| andrewbogott | Kiall: please bear in mind that you have at least one actual user running a production install, who may enjoy upgrading from time to time. | 20:46 |
| Kiall | grab the `show processlist` output from mysql, if it's locking on the DB, that should show something | 20:46 |
| andrewbogott | ok, let me see if we captured that last time... | 20:46 |
| *** krotscheck has joined #openstack-dns | 20:46 | |
| *** openstackgerrit has quit IRC | 20:47 | |
| *** openstackgerrit has joined #openstack-dns | 20:47 | |
| *** ChanServ sets mode: +v openstackgerrit | 20:47 | |
| andrewbogott | Kiall: this may be of interest: https://phabricator.wikimedia.org/T124680#2007970 | 20:48 |
| Kiall | Also, you said you had exactly 2 nameservers, 1 active and 1 standby - if you can set master=false, slave=false on the standby - with a giant caveat that it needs to be flipped to true to use the standby - and see if the lockups happen, that would rule out the 2 locking on each other. | 20:48 |
| andrewbogott | Unfortunately the lockups happen only every few weeks. So it’s very hard to experiment with any reliability. | 20:50 |
| andrewbogott | Would you expect the systems to typically be master/slave true/true? Or false/true? | 20:50 |
| Kiall | the worst kind. From that URL, I see a comment at the end re "Does pdns delete all records and reimport them periodically (see rows_inserted and rows_deleted)?" | 20:50 |
| andrewbogott | Kiall: the answer to that is ‘yes’ isn’t it? Basically that happens every time there’s a sync | 20:51 |
| Kiall | Like 142nd disconnect. This happening as often to others? | 20:52 |
| Kiall | It indeed does, it starts a TX, deletes all the records in the zone, and repopulates the contents before completing the TX | 20:52 |
| *** james_li has quit IRC | 20:53 | |
| Kiall | master=false for most deployments | 20:53 |
| Kiall | slave=true for all (unless you're trying to debug things) | 20:54 |
| andrewbogott | ok, that sounds right to me | 20:54 |
| Kiall | slave=false will mean it ignores designate-mdns when it's told to update content | 20:54 |
| *** james_li has joined #openstack-dns | 20:54 | |
| andrewbogott | Kiall: if I paste a redacted version of my designate config would you be willing to take a look? There may be obvious mistakes. | 20:54 |
| *** eandersson_ has joined #openstack-dns | 20:55 | |
| Kiall | Yea, no problem... be consistent in redacting tho ;) e.g. 1.0.0.1 -> x.x.x.x, 1.0.0.2 -> y.y.y.y - it can be hard to trace when everything is *'s ;) | 20:55 |
| andrewbogott | ok, will ping you shortly | 20:56 |
| eandersson_ | Kiall: Do you know why records aren't deleting properly after an upgrade from Kilo to Liberty? | 20:56 |
| eandersson_ | mugsie, thought that it could be a bug | 20:57 |
| eandersson_ | or sounded like it at least | 20:57 |
| eandersson_ | Any major changes I missed maybe during the upgrade. | 20:57 |
| Kiall | ehh, records not deleting? but new ones do get created? that's a new one for me... | 20:57 |
| eandersson_ | Stuck pending in the DB | 20:58 |
| eandersson_ | Feels like I ran into a similar issue when upgrading to Kilo. | 20:58 |
| eandersson_ | I am also seeing the following when using the cli http://paste.openstack.org/show/487796/ | 20:59 |
| *** Kiall has quit IRC | 21:00 | |
| *** Kiall has joined #openstack-dns | 21:00 | |
| *** Kiall has quit IRC | 21:00 | |
| *** Kiall has joined #openstack-dns | 21:01 | |
| Kiall | eandersson_: so Q, things only get stuck in pending when you're deleting a record? other actions do work? | 21:02 |
| Kiall | (or, mugsie will prob be back in a bit! might be better than starting over :D) | 21:02 |
| eandersson_ | I only tried adding / removing records so far | 21:03 |
| eandersson_ | Yea, you are probabl right. | 21:03 |
| eandersson_ | Was going to deal with this tomorrow, but... I have a difficult time leaving something broken lol | 21:03 |
| mugsie | eandersson_: hey - back | 21:09 |
| *** Kiall has quit IRC | 21:09 | |
| mugsie | stupid bus driver took the worng bus from the depot and we had to go back and swap buses -_- | 21:09 |
| eandersson_ | haha that stinks | 21:10 |
| *** Kiall has joined #openstack-dns | 21:10 | |
| eandersson_ | So mugsie I upgraded to trunk, still nothing. | 21:10 |
| andrewbogott | Kiall: behold, my crufty config: https://phabricator.wikimedia.org/P2650 | 21:10 |
| eandersson_ | sorry not trunk, but latest stable/liberty | 21:10 |
| mugsie | ok. do you have logs from pool-manager / minidns that show any issues? | 21:11 |
| Kiall | andrewbogott: double checking, kilo? | 21:11 |
| mugsie | I am just about to hop on a call, will be back in a few | 21:11 |
| andrewbogott | Yes, kilo. And a lot of stuff in there is supporting custom sink handlers; should be obvious which. | 21:11 |
| Kiall | L57 can be removed, no longer used.. (Just saying while I see it) | 21:12 |
| andrewbogott | Kiall: that file was migrated from older versions, so if you see settings that are deprecated let me know and I’ll snip them out | 21:12 |
| Kiall | :D | 21:13 |
| andrewbogott | heh, you’re on it :) | 21:13 |
| Kiall | L143 looks dud aswell.. | 21:14 |
| *** mlavalle has quit IRC | 21:14 | |
| Kiall | OMG. why FreeNode/my ISP, why do you hate me? | 21:15 |
| andrewbogott | so [service:pool_manager] can be 100% defaults? | 21:15 |
| eandersson_ | andrewbogott, you upgrading to kilo? | 21:16 |
| Kiall | I'd have expected to see a second [pool_nameserver:*] section, for the standby pDNS. | 21:16 |
| Kiall | and is mdns running on both the pDNS servers? if so, the masters = 127.0.0.1:5354 is fine.. but you can list out non-loopback IPs for them all there too.. No harm though | 21:17 |
| Kiall | re all defaults, yea.. it can be | 21:17 |
| andrewbogott | ok, now this is getting interesting :) | 21:17 |
| andrewbogott | I have two separate boxes, one is running mdns and pdns, the other (the secondary) running only pdns | 21:17 |
| andrewbogott | the second had up-to-date records last I checked, but I haven’t checked in a while | 21:18 |
| Kiall | it's getting them via the first one doing the AXFR | 21:18 |
| andrewbogott | ah,ok, so right now my secondary is a slave of my primary rather than a slave of mdns | 21:18 |
| andrewbogott | (if I understand my terminology correctly) | 21:18 |
| Kiall | But, if it obtains a lock on the DB and tries to AXFR, it might just hold the lock for quite some time | 21:18 |
| Kiall | kinda.. it's not slaving from the primary pDNS, it just happens to be sharing a datastore so "flukes" into getting content | 21:19 |
| andrewbogott | Kiall: so, would it be an option to just configure my secondary to be master=false slave=false, and rely on it getting what it needs from the db that’s written to by the primary? | 21:19 |
| andrewbogott | in other words, declare that ‘fluke’ to be on purpose :) | 21:20 |
| Kiall | That's an option, until the primary falls over. | 21:20 |
| andrewbogott | ah, right, of course, then it will drift out of sync. | 21:20 |
| Kiall | and it may very well be that pDNS is very eager to obtain a lock, meaning the second one not being able to AXFR could do weird things. | 21:20 |
| andrewbogott | So, action items: | 21:21 |
| andrewbogott | 1) Add a pool_nameserver entry for secondary pdns | 21:21 |
| andrewbogott | 2) somethingsomething to make mdns send xfr updates to both | 21:21 |
| andrewbogott | what does 2) look like? is it just | 21:22 |
| andrewbogott | masters = <primaryip>:5354, <secondaryip>:5454 ? | 21:22 |
| Kiall | yep, and 1 more thing to complete #2.. 1 sec while I dig it out | 21:22 |
| andrewbogott | wait... | 21:23 |
| andrewbogott | masters should point to mdns, correct? | 21:23 |
| andrewbogott | so it should actually be | 21:24 |
| Kiall | Yes, points to mDNS - ideally more than 1 mDNS so if it crashes, it still OK | 21:24 |
| andrewbogott | masters = <mdnsip>:5354 | 21:24 |
| andrewbogott | in both sections | 21:24 |
| andrewbogott | (and, yeah, with a second mdns ideally) | 21:24 |
| andrewbogott | ok, I think I follow | 21:24 |
| Kiall | and.. UPDATE pdns.domains SET masters="mDNSip:port,mDNSip:port"; to update existing domains in the powerdns DB. | 21:24 |
| Kiall | UPDATE pdns.domains SET master="mDNSip:port,mDNSip:port"; * | 21:25 |
| andrewbogott | ok. In the short run I’m not expecting to add a second mdns, I’ll just point both pdns servers to the existing mdns | 21:25 |
| Kiall | (We tend to err on the side of not updating existing things like ^, as you might have millions of zones) | 21:25 |
| andrewbogott | so in that case that UPDATE is not needed, correct? | 21:25 |
| Kiall | It is, as it'll have 127.0.0.1 in there right now | 21:26 |
| andrewbogott | hm... | 21:26 |
| andrewbogott | So, sorry, I’m going to make you repeat yourself I think. ‘master’ is conveyed to pdns, and pdns is told “listen for updates from this" | 21:27 |
| andrewbogott | is that right? | 21:27 |
| andrewbogott | oh, sorry, no | 21:28 |
| andrewbogott | you just told me — I’m conveying that to pdns myself, via that update command | 21:28 |
| Kiall | It's 2 way.. designate will shout over to powerdns saying "Hey, I think I have some new content for you - foo.com, serial 1234".. powerdns will check it's DB to see if it has foo.com 1234, and if not, reaches back out to one of the mdns servers asking for the full content of the zone | 21:28 |
| andrewbogott | ok, I’m following I think. | 21:29 |
| andrewbogott | Kiall: any other points of pain in that config? | 21:31 |
| andrewbogott | (And, I don’t rightly remember where those uuids come from, they only have to be internally consistent within the config, right? Or are they in a db someplace?) | 21:32 |
| Kiall | Yes, Interally consistent. | 21:32 |
| Kiall | And, no - no other weirdness that I see.. Though, I am curious what the nova_ldap handler is? | 21:33 |
| andrewbogott | Kiall: It’s something I’m trying to kill :) | 21:33 |
| eandersson_ | That was the first thing I implemented as well. =] | 21:34 |
| andrewbogott | But, for the moment, we have per-instance ldap entries that store various things, most interestingly puppet node definitions. | 21:34 |
| andrewbogott | Sink seemed like a reasonable place to stow that since it’s getting notifications for instance creation and knows all the right things anyway. | 21:34 |
| Kiall | Hah, fair enough :) | 21:35 |
| andrewbogott | Kiall: thank you for looking over my config! I may solicit a code review when I have the config patch written. | 21:35 |
| Kiall | sure.. no problem | 21:36 |
| openstackgerrit | Chris Johnson proposed openstack/designate: Cleanup after enable files refactor https://review.openstack.org/283268 | 21:36 |
| Kiall | Hopefully that really was it, if the standby pDNS was somehow holding the lock while it tried to talk to a master that didn't actually exist (127.0.0.1..), it's certainly possible it's caused things to deadlock. But.. We'll see if the happens again after those config changes | 21:37 |
| Kiall | gotta run to the shops before they chose.. back in a bit. | 21:38 |
| *** porunov has joined #openstack-dns | 21:43 | |
| mugsie | eandersson_: hey - sorry, my boss likes to talk... a lot | 21:43 |
| mugsie | :) | 21:43 |
| * mugsie just checked he wasnt in here | 21:44 | |
| eandersson_ | haha no worries | 21:44 |
| eandersson_ | I'll have to pick this up tomorrow though | 21:44 |
| mugsie | cool. just give me a ping | 21:44 |
| mugsie | I am on most of the time :) | 21:44 |
| eandersson_ | Is there anything that needs to be changed in the config after an upgrade from Kilo to Liberty? | 21:44 |
| eandersson_ | Anything obvious that I might have missed? | 21:44 |
| eandersson_ | Since you probably wont be online until late for me tomorrow =] | 21:46 |
| mugsie | well, I am UTC, so I should be OK for you (in theory) :). Kilo to liberty - let me have a quick look | 21:47 |
| eandersson_ | oh wow working late? :D | 21:48 |
| mugsie | just a bit | 21:48 |
| mugsie | the sample config hasnt changed | 21:49 |
| mugsie | (much) - nothing of major interest | 21:49 |
| mugsie | there should not be anything extra needed | 21:49 |
| eandersson_ | Is the new service required? | 21:55 |
| Kiall | zone-manager? Nope, totally optional for some new features... | 21:56 |
| eandersson_ | ah kk didn't read anything about it *yet* | 21:57 |
| eandersson_ | I'll go and play some Overwatch then and try again tomorrow =] | 21:58 |
| eandersson_ | thanks guys for the help | 21:58 |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Update Bind 9 backend and other documentation https://review.openstack.org/282437 | 22:05 |
| openstackgerrit | Tim Simmons proposed openstack/designate: Add an on-demand single-target sync method https://review.openstack.org/274292 | 22:22 |
| *** jordanP has joined #openstack-dns | 22:27 | |
| *** ducttape_ has quit IRC | 22:40 | |
| *** ducttape_ has joined #openstack-dns | 22:42 | |
| andrewbogott | Kiall: at your leisure, https://gerrit.wikimedia.org/r/#/c/271797/ | 22:54 |
| andrewbogott | (you surely cannot leave comments there, but the patch itself should be public) | 22:54 |
| *** jordanP has quit IRC | 22:57 | |
| openstackgerrit | Merged openstack/designate: Fix a missing '\' in doc https://review.openstack.org/281359 | 23:03 |
| *** eandersson_ has quit IRC | 23:07 | |
| *** jmcbride has quit IRC | 23:17 | |
| *** f1ller is now known as filler | 23:19 | |
| *** ducttape_ has quit IRC | 23:20 | |
| *** pglass has quit IRC | 23:21 | |
| *** km has joined #openstack-dns | 23:22 | |
| *** kei_yama has joined #openstack-dns | 23:27 | |
| Kiall | andrewbogott: just heading to bed, but had a real quick look.. So, it should be 2 pool nameserver sections, and 1 pool target section.. | 23:36 |
| Kiall | If you move forward with 2 pdns and 2 DBd, then it's 2 nameserver sections and 2 target sections.. | 23:37 |
| andrewbogott | so target == mdns | 23:37 |
| andrewbogott | and nameserver == pdns? | 23:38 |
| andrewbogott | Kiall: does that mean that I’m setting a url to the pdns database for mdns and /not/ for pdns? That confuses me :) | 23:39 |
| Kiall | target is where creates and deletes go, nameservers are where we send notifies to and check that changes have gone live (for powerdns..) | 23:39 |
| Kiall | E.g. creating a zone with a shared DB only needs to happen once, while with 2 DBs it needs to happen twice. | 23:40 |
| andrewbogott | I follow, it’s just that in the ‘before’ version of the config that you looked at before... | 23:41 |
| andrewbogott | under [pool_target:f26e0b32-736f-4f0a-831b-039a415c481e] | 23:41 |
| Kiall | The main thing I think will help your issue is the correction of the masters line | 23:41 |
| andrewbogott | we had a connection type = powerdns set | 23:41 |
| andrewbogott | oh, and of course... | 23:42 |
| andrewbogott | pdns is both a target /and/ a nameserver | 23:42 |
| andrewbogott | because of the mysql +also axfr thing | 23:42 |
| andrewbogott | I’m slow to grasp that, it seems | 23:42 |
| andrewbogott | I’ll rewrite, and (maybe) ping you tomorrow. Have a good night! | 23:42 |
| Kiall | Kinda, the DB is the target, powerdns is the nameserver.. It's odd, I know.. It's hard to map the many different ways different DNS servers work into something that's obvious for all of them! | 23:43 |
| andrewbogott | ‘db as target’ makes sense | 23:44 |
| * andrewbogott comments accordingly | 23:44 | |
| Kiall | Anyway, gone.. I'll be around tomorrow anyway.. | 23:44 |
| Kiall | Cya | 23:44 |
| *** rsyed is now known as rsyed_away | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
