| *** ducttape_ has joined #openstack-dns | 00:54 | |
| *** ChanServ changes topic to "OpenStack Designate - Logged @ http://eavesdrop.openstack.org/irclogs/%23openstack-dns" | 00:55 | |
| -openstackstatus- NOTICE: Gerrit has been downgraded to version 2.8 due to the issues observed today. Please report further problems in #openstack-infra. | 00:55 | |
| *** km__ has joined #openstack-dns | 01:03 | |
| *** km has quit IRC | 01:03 | |
| *** jmcbride has joined #openstack-dns | 01:39 | |
| *** jmcbride has quit IRC | 02:02 | |
| *** km has joined #openstack-dns | 02:15 | |
| *** km__ has quit IRC | 02:16 | |
| *** richm has quit IRC | 02:18 | |
| *** ducttape_ has quit IRC | 02:22 | |
| *** ducttape_ has joined #openstack-dns | 02:46 | |
| *** ducttape_ has quit IRC | 02:46 | |
| *** EricGonczer_ has joined #openstack-dns | 02:51 | |
| *** EricGonczer_ has quit IRC | 03:36 | |
| *** EricGonczer_ has joined #openstack-dns | 03:44 | |
| *** EricGonczer_ has quit IRC | 03:58 | |
| *** EricGonczer_ has joined #openstack-dns | 04:06 | |
| *** EricGonczer_ has quit IRC | 04:16 | |
| *** stanzgy has joined #openstack-dns | 04:19 | |
| *** nihilifer has joined #openstack-dns | 06:42 | |
| *** openstackstatus has quit IRC | 07:52 | |
| *** openstack has joined #openstack-dns | 07:53 | |
| *** ChanServ sets mode: +v openstack | 07:53 | |
| *** jordanP has joined #openstack-dns | 08:10 | |
| openstackgerrit | stanzgy proposed openstack/designate: Allow sink to create multiple entries per instance https://review.openstack.org/170302 | 08:40 |
|---|---|---|
| openstackgerrit | Endre Karlson proposed openstack/python-designateclient: V2 CLI Support https://review.openstack.org/133676 | 08:56 |
| openstackgerrit | stanzgy proposed openstack/designate: Set cfg.pool_target.options as secret https://review.openstack.org/182214 | 09:47 |
| *** eandersson has joined #openstack-dns | 10:32 | |
| *** stanzgy has quit IRC | 10:37 | |
| *** untriaged-bot has joined #openstack-dns | 11:02 | |
| untriaged-bot | Untriaged bugs so far: | 11:02 |
| untriaged-bot | https://bugs.launchpad.net/designate/+bug/1453591 | 11:02 |
| openstack | Launchpad bug 1453591 in Designate "Identity v3 PKI token over 8k does not work" [Undecided,Fix committed] - Assigned to Dag Stenstad (dag-stenstad) | 11:02 |
| untriaged-bot | https://bugs.launchpad.net/designate/+bug/1454175 | 11:02 |
| openstack | Launchpad bug 1454175 in Designate "pool-manager configuration options will leak db passwd" [Undecided,In progress] - Assigned to stanzgy (stanzgy) | 11:02 |
| untriaged-bot | https://bugs.launchpad.net/designate/+bug/1453937 | 11:02 |
| openstack | Launchpad bug 1453937 in Designate "designate dashboard tox tests are not passing" [Undecided,In progress] - Assigned to Eric Peterson (ericpeterson-l) | 11:02 |
| untriaged-bot | https://bugs.launchpad.net/designate/+bug/1453870 | 11:02 |
| openstack | Launchpad bug 1453870 in Designate "designate dashboard create domain should honor quotas" [Undecided,In progress] - Assigned to Eric Peterson (ericpeterson-l) | 11:02 |
| *** untriaged-bot has quit IRC | 11:02 | |
| *** stanzgy_ has quit IRC | 11:14 | |
| *** stanzgy_ has joined #openstack-dns | 11:14 | |
| openstackgerrit | Merged openstack/designate: Set cfg.pool_target.options as secret https://review.openstack.org/182214 | 11:22 |
| openstackgerrit | Merged openstack/designate: Allow sink to create multiple entries per instance https://review.openstack.org/170302 | 11:22 |
| openstackgerrit | Merged openstack/designate: updating tox pep8 issues https://review.openstack.org/182056 | 11:22 |
| *** km has quit IRC | 11:29 | |
| openstackgerrit | Graham Hayes proposed openstack/designate-specs: Add graded backends https://review.openstack.org/169859 | 11:40 |
| eandersson | timsim: I was wrong yesterday about the recordsetnot found, was not caused by create -> delete -> create. It seems to happen even with a simple create. | 11:43 |
| eandersson | What does create recordset depend on besides central? | 11:43 |
| *** EricGonczer_ has joined #openstack-dns | 11:53 | |
| *** harmw has quit IRC | 12:00 | |
| *** harmw has joined #openstack-dns | 12:01 | |
| *** harmw has quit IRC | 12:02 | |
| *** harmw has joined #openstack-dns | 12:02 | |
| eandersson | I am running into major issues when running two control nodes on a site. | 12:09 |
| eandersson | If I disable ccn01, and run only ccn02 everything works fine. | 12:09 |
| eandersson | but if I have both about 50% of all create requests generate a recordsetnot found error | 12:10 |
| eandersson | configs are identical on both ccn0[12] | 12:10 |
| *** ducttape_ has joined #openstack-dns | 12:10 | |
| mugsie | eandersson: control node? you mean central? | 12:11 |
| eandersson | Yea | 12:12 |
| eandersson | well central, mdns, pool-manager | 12:12 |
| mugsie | hummm | 12:12 |
| mugsie | let boot up a few things and see what happens | 12:13 |
| eandersson | I am running stable/kilo from 30th of April | 12:13 |
| eandersson | The only error I see in the central logs is Expected exception during message handling () _dispatch_and_reply /usr/local/designate/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145 | 12:15 |
| Kiall | eandersson: Q - are the 2 designate-central's pointed at the same DB server? | 12:18 |
| eandersson | yep | 12:19 |
| eandersson | vip though | 12:20 |
| Kiall | Multiple DB servers behind the scene? are they by any chance standard asynchronous mysql replication? | 12:21 |
| eandersson | should be master/slave setup | 12:21 |
| eandersson | but need to confirm that | 12:21 |
| Kiall | Also - Can you remove the expected = True from the recordsetnotfound exepction in exceptions.py, and reproduce? You should then get a stacktrack telling you where exactly where the exception is raised | 12:21 |
| eandersson | ah perfect sure | 12:22 |
| Kiall | and - possibly a bunch of other stack traces that are "normal" and aren't a real issue | 12:22 |
| Kiall | But there should be at least 1 with the actual trace we want.. | 12:22 |
| openstackgerrit | Merged openstack/designate: Denominator Agent https://review.openstack.org/173472 | 12:24 |
| *** ducttape_ has quit IRC | 12:31 | |
| *** ryanpetrello has joined #openstack-dns | 12:37 | |
| eandersson | http://paste.openstack.org/show/FkrF6wyChNsX73uMPWIs/ | 12:39 |
| eandersson | It always works on the second try. | 12:39 |
| eandersson | I'll need to confirm the mysql setup though. | 12:39 |
| openstackgerrit | Merged openstack/designate: Add quota check for domain create https://review.openstack.org/182098 | 12:50 |
| openstackgerrit | Ryan Petrello proposed openstack/designate: Remove unnecessary RestController usage. https://review.openstack.org/182273 | 12:54 |
| ryanpetrello | anybody around willing to help me push ^ through? | 12:54 |
| ryanpetrello | I'm fixing a bug in RestController in pecan, and ^ is necessary to get the designate tests passing afterwards | 12:54 |
| mugsie | ryanpetrello: Will look now | 12:55 |
| *** EricGonczer_ has quit IRC | 12:56 | |
| mugsie | ryanpetrello: my reading of the linked patch is the following - if a sub controller has a PUT, but its parent does not, the routing will fail? | 12:57 |
| Kiall | eandersson: sorry, wasn't paying attention to IRC.. let me look | 12:57 |
| mugsie | or is it just controllers with no methods at all will fail to pass requests to sub controllers? | 12:58 |
| ryanpetrello | the change RestController will now raise 405 in situations where it previously raised 404's | 12:58 |
| ryanpetrello | and it will specify an Allow header with allowed methods | 12:58 |
| ryanpetrello | the trouble is that one designate test asserts a 404 response | 12:59 |
| ryanpetrello | after the pecan change, it will be 405 | 12:59 |
| ryanpetrello | but I noticed that the designate controller in question doesn't need to be a RestController anyways | 12:59 |
| ryanpetrello | another approach I could take | 13:00 |
| ryanpetrello | would be to leave the RestController and update the test to be e.g., | 13:00 |
| mugsie | will ^ patch give a 405 is there is no GET method, but there is say a POST ? | 13:00 |
| ryanpetrello | status in (404, 405) | 13:00 |
| ryanpetrello | (yes) | 13:00 |
| ryanpetrello | the idea is that if a request is made where an alternative exists, it'll give a 405 (rather than a 404) | 13:00 |
| mugsie | sweet | 13:01 |
| mugsie | there is a few hacks across our code base to do the 405 vs 404 | 13:01 |
| ryanpetrello | I just need to update designate in the mean time | 13:01 |
| ryanpetrello | because pecan gates on it | 13:01 |
| ryanpetrello | (to merge my change) | 13:01 |
| ryanpetrello | kind of a chicken-and-egg problem | 13:01 |
| mugsie | cool. as soon as tests come back I will +2 | 13:01 |
| ryanpetrello | cool thx | 13:01 |
| mugsie | np | 13:02 |
| ryanpetrello | yea, I'm betting you all can lose some code after this merges in pecan | 13:02 |
| mugsie | heres hoping | 13:02 |
| mugsie | https://github.com/openstack/designate/blob/master/designate/api/v2/controllers/zones/tasks/abandon.py#L44 <- really want rid of that for example | 13:03 |
| ryanpetrello | yep, this should allow you to get rid of that | 13:05 |
| ryanpetrello | it's exactly what it's for | 13:05 |
| *** richm has joined #openstack-dns | 13:06 | |
| *** nihilifer has quit IRC | 13:13 | |
| Kiall | eandersson: Humm, I'm not sure I see where the issue is :( | 13:15 |
| Kiall | eandersson: It really feels like async DB replication meaning data we just wrote isn't available for reading.. But it's a total hunch on my part | 13:15 |
| *** richm1 has joined #openstack-dns | 13:23 | |
| *** richm has quit IRC | 13:24 | |
| mugsie | ryanpetrello: seems to have a pep8 issue -_- | 13:28 |
| ryanpetrello | blargh | 13:28 |
| ryanpetrello | okay, I'll get it fixed | 13:28 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Remove unnecessary RestController usage https://review.openstack.org/182273 | 13:28 |
| mugsie | i think kiall just fixed the commit message | 13:28 |
| ryanpetrello | haha, the commit message | 13:28 |
| ryanpetrello | yep | 13:28 |
| Kiall | ryanpetrello: yea, just edited it real quick there are +2'd ;) | 13:29 |
| ryanpetrello | awesome, thanks :D | 13:29 |
| Kiall | np | 13:29 |
| Kiall | timsim: ping me when your about | 13:30 |
| *** richm1 has quit IRC | 13:31 | |
| *** richm has joined #openstack-dns | 13:35 | |
| *** andrewbogott_afk is now known as andrewbogott | 14:00 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Fix check+set race condition in APIv1/Sink https://review.openstack.org/182303 | 14:00 |
| *** jmcbride has joined #openstack-dns | 14:05 | |
| *** richm has quit IRC | 14:10 | |
| openstackgerrit | Eric Larson proposed openstack/designate: Add the Hook Point API https://review.openstack.org/164748 | 14:11 |
| openstackgerrit | Merged openstack/designate: Remove unnecessary RestController usage https://review.openstack.org/182273 | 14:14 |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add Akamai Management comamnds https://review.openstack.org/182309 | 14:14 |
| *** EricGonczer_ has joined #openstack-dns | 14:17 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add Akamai Management comamnds https://review.openstack.org/182309 | 14:18 |
| *** richm has joined #openstack-dns | 14:31 | |
| *** jmcbride has quit IRC | 14:42 | |
| *** ducttape_ has joined #openstack-dns | 14:44 | |
| *** pglass has joined #openstack-dns | 14:45 | |
| *** jmcbride has joined #openstack-dns | 14:46 | |
| timsim | Kiall: ping | 14:47 |
| Kiall | timsim: heya, so.. wanted to check if I gave enough detail in the review comment re eventlet.tpool for the import stuff? | 14:48 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Fix check+set race condition in APIv1/Sink https://review.openstack.org/182303 | 14:49 |
| timsim | I suppose 'use eventlet's tpool' is probably sufficient. I've never used that before, but I'm assuming it's going to create it's own threadpool to use? What's the advantage there vs using the regular threadgroup? Is it a separate process? | 14:50 |
| Kiall | Well, the threadgroup is eventlet greenthreads, while eventlet.tpool is real OS threads.. | 14:50 |
| Kiall | Which.. Now that I sat it, seems backwards, but it's not ;) | 14:51 |
| timsim | Ahh lovely. Ok cool. I'll do that. | 14:52 |
| *** vinod has joined #openstack-dns | 14:55 | |
| Kiall | s/sat/say/ | 14:56 |
| *** jmcbride has quit IRC | 14:58 | |
| *** jmcbride has joined #openstack-dns | 14:58 | |
| openstackgerrit | Vinod Mangalpally proposed openstack/designate: Add some more settings to the sample conf file https://review.openstack.org/181557 | 15:01 |
| *** richm has quit IRC | 15:05 | |
| *** richm has joined #openstack-dns | 15:08 | |
| *** nihilifer has joined #openstack-dns | 15:09 | |
| *** ericpeterson has joined #openstack-dns | 15:14 | |
| openstackgerrit | Eric Larson proposed openstack/designate-specs: Add hook-point-api specification and liberty specs https://review.openstack.org/171689 | 15:17 |
| *** ducttape_ has quit IRC | 15:18 | |
| *** nihilifer has quit IRC | 15:18 | |
| *** shakamunyi has joined #openstack-dns | 15:22 | |
| *** barra204 has joined #openstack-dns | 15:22 | |
| *** ducttape_ has joined #openstack-dns | 15:38 | |
| *** ericpeterson has quit IRC | 15:42 | |
| *** nihilifer has joined #openstack-dns | 15:46 | |
| *** nihilifer has quit IRC | 15:50 | |
| *** jhazen has joined #openstack-dns | 15:56 | |
| *** openstackgerrit_ has quit IRC | 16:11 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add Akamai Management comamnds https://review.openstack.org/182309 | 16:13 |
| *** chirag789 has joined #openstack-dns | 16:35 | |
| chirag789 | Hi, I am using designate and after following the documentation available online, I was able to use v1. Question is : v2 is available to use including recordset? do we have a documentation on v2 rest api. I am pretty sure the documentation available designate.readthedocs.org/en/latest/getting-started.html is outdated. | 16:37 |
| timsim | chirag789: The docs now live at http://docs.openstack.org/developer/designate/ | 16:39 |
| timsim | The API documentation is here: http://docs.openstack.org/developer/designate/rest.html including the v2 api. | 16:39 |
| timsim | I think it's on someones list to get the old docs to redirect to the new ones. | 16:40 |
| Kiall | timsim: Read the docs - 100% gone now! | 16:54 |
| ekarlso | Kiall: shall I delete them ? | 16:55 |
| Kiall | ekarlso: I did | 16:55 |
| timsim | woot! | 16:55 |
| Kiall | Another week or two and google will stop sending people there! | 16:55 |
| timsim | Are these pages just cached or something http://designate.readthedocs.org/en/latest/getting-started.html | 16:56 |
| ekarlso | ok Kiall :D | 16:56 |
| Kiall | timsim: yea, I guess so | 16:56 |
| timsim | Cool. Glad it's gone. | 16:57 |
| Kiall | "You don't have any projects yet, but you can star".. when I used to have designate listed on that page ;) | 16:57 |
| timsim | Cool cool | 16:57 |
| tg90nor | hi, i have configured designate according to http://docs.openstack.org/developer/designate/install/ubuntu-dev.html , but when i try to start designate-pool-manager, i get the error CRITICAL designate [-] NoPoolTargetsConfigured | 16:57 |
| tg90nor | am i missing something? | 16:57 |
| ekarlso | tg90nor: ya, u need to configure a pool! | 16:58 |
| tg90nor | how? | 16:58 |
| ekarlso | timsim: isn't there docs for that ? | 16:58 |
| timsim | tg90nor: It's in the sample config here: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L274 | 16:58 |
| Kiall | Ah, right. Did we not .. ehh.. update the config docs for Kilo? :/ | 16:58 |
| ekarlso | timsim: any api parts needed for that ? | 16:58 |
| timsim | But I don't think it's made into the proper getting started guide yet. | 16:58 |
| ekarlso | Kiall: nope D | 16:59 |
| tg90nor | timsim: thx | 16:59 |
| timsim | You bet | 16:59 |
| openstackgerrit | Endre Karlson proposed openstack/designate: Add Akamai Management comamnds https://review.openstack.org/182309 | 17:01 |
| *** ducttape_ has quit IRC | 17:02 | |
| *** stanzgy_ has quit IRC | 17:02 | |
| *** Guest25580 is now known as dan | 17:02 | |
| *** stanzgy_ has joined #openstack-dns | 17:02 | |
| *** untriaged-bot has joined #openstack-dns | 17:02 | |
| untriaged-bot | No untriaged bugs so far! \o/ | 17:02 |
| *** untriaged-bot has quit IRC | 17:02 | |
| *** ducttape_ has joined #openstack-dns | 17:03 | |
| openstackgerrit | Tim Simmons proposed openstack/designate: Asynchronous Zone Import https://review.openstack.org/181528 | 17:03 |
| timsim | Kiall: ^ works great when there's only one import, but when there are concurrent ones, thread ownership seems to get confused and they can't release locks. | 17:03 |
| mugsie | timsim: thats not good enogh | 17:12 |
| mugsie | :P | 17:12 |
| *** jordanP has quit IRC | 17:13 | |
| *** Timsimphone has joined #openstack-dns | 17:27 | |
| *** Timsimphone has quit IRC | 17:28 | |
| *** eandersson has quit IRC | 17:31 | |
| Kiall | timsim: humm, not 100% sure I understand what you mean? | 17:37 |
| chirag789 | wow thanks. Thank for the docs. | 17:37 |
| chirag789 | Another thing. I was reading the blueprints and meeting logs. Seems like you are planning to add filtering api and planning to remove records and v1 api? | 17:38 |
| mugsie | chirag789: yes - v1 has been marked as depricated | 17:39 |
| mugsie | there is some filtering on the v2 API currently | 17:39 |
| chirag789 | Okay. I will move to v2 then. Lastly, I am new to OS and designate. Thinking to attend weekly meetings. On this page it shows March 11 https://wiki.openstack.org/wiki/Meetings/Designate . Can you please provide me information on when there will be next meeting. | 17:41 |
| mugsie | chirag789: they are every wednesday at 5pm UTC | 17:45 |
| mugsie | but next week, there won't be a meeting, as most of us will be in Vancouver at the summit | 17:45 |
| *** chlong has quit IRC | 17:46 | |
| openstackgerrit | Ron Rickard proposed openstack/designate-specs: WIP: Blueprint for Transaction Support https://review.openstack.org/180784 | 17:49 |
| chirag789 | okay. Thanks. Also, I have a question in terms of how designate works. For example, we have an openstack with two tenants "xyz" and "abc". Openstack has keystone domain "default". This domain is not related to Designate domain at all i believe. Now, as per my understanding, if admin (openstack admin) or any one in admin tenant is creating domains on designate, lets say, "www.xyz.com" for xyz tenant and "www.abc.com" for abc | 17:52 |
| chirag789 | If users under xyz and abc tenants tries to create record, it will fail and will throw error that you don't have access to create the record (something like that). So, what can I do to allow users under tenants to create the record. I tried in v1 and it was giving the error. | 17:53 |
| *** chlong has joined #openstack-dns | 17:59 | |
| timsim | Kiall: http://logs.openstack.org/28/181528/3/check/gate-designate-dsvm-bind9/2a6327b/logs/screen-designate-central.txt.gz#_2015-05-12_17_28_57_298 Down at the bottom there. Same error happens here: http://logs.openstack.org/28/181528/3/check/gate-designate-python27/50b35d7/testr_results.html.gz I have no idea why all the tempest tests failed, they all passed for me locally. | 18:01 |
| Kiall | Ah, ehh... I've not seen that before | 18:02 |
| Kiall | https://bitbucket.org/eventlet/eventlet/pull-request/29/fix-use-of-semaphore-with-tpool-issue-137/diff | 18:02 |
| Kiall | timsim: based on my reading of that pull request, it's *probably* caused by calling self.update_zone_import(context, zone_import) inside the thread.. | 18:04 |
| timsim | Not create_domain? | 18:04 |
| Kiall | Well - that too I suppose | 18:04 |
| Kiall | So, Q - Where is the bulk of the time spent? | 18:05 |
| Kiall | Between L2409 and L2502, or elsewhere? | 18:05 |
| timsim | The vast majority of the time is spent creating the recordsets/records in storage | 18:06 |
| *** jmcbride has quit IRC | 18:06 | |
| timsim | But with a large file, there would be significant time spent between 2490 and 2502. | 18:07 |
| Kiall | Let me rephrase, it case it changes it answer ;).. Where is the bulk of the time spent while central is locked up? I have a suspicion the lockup is caused by 2940 -> 2502, and not the call to storage (since eventlet is doing it's thing there) | 18:07 |
| timsim | Yeah, that's true. I've seen, sometimes, the API return when it gets to create_domain | 18:08 |
| timsim | where it would otherwise timeout, even if everything runs smoothly. | 18:08 |
| timsim | But I think central can get locked up doing that big domain create as well | 18:09 |
| timsim | Yeah, I can't GET the imports/id while the domain is being created in central, if it's all in a regular greenthread | 18:14 |
| Kiall | sorry - checking the code for something related.. 1 sec | 18:14 |
| Kiall | re "Yeah, I can't GET the imports/id while the domain is being created in central, if it's all in a regular greenthread" | 18:16 |
| Kiall | that's likely because of the transaction decorator, the TX is being kept open for the whole duration of the import as is.. | 18:17 |
| Kiall | And the commit doesn't happen until right at the end | 18:17 |
| timsim | Oh derp, thought I removed that. | 18:17 |
| Kiall | Can you trove moving only 2940 -> 2502 into the tpool execute, workers = 1, and see if you get lockups during an import? | 18:20 |
| Kiall | and no @transaction decorator - I suspct that'll end up doing what we need | 18:20 |
| Kiall | brb | 18:20 |
| Kiall | timsim: BTW, re moving just the bit into the thead, tpool.execute() will return the return value of whatever you execute in the thread.. | 18:27 |
| Kiall | (At least, the code looks like it does ;)) | 18:27 |
| timsim | yep, just about there | 18:27 |
| timsim | I don't think you can pass workers=1 to execute though | 18:27 |
| Kiall | Oh, i mean 1 designate-central worker | 18:27 |
| timsim | Ah ok. Makes sense | 18:28 |
| timsim | Still can't 'GET /v2/zones/tasks/imports/566f5e40-fe42-4612-8740-eb6ca3e969ec' until create_domain completes. http://paste.openstack.org/show/221074/ | 18:30 |
| * Kiall hates eventlet sometimes | 18:31 | |
| Kiall | timsim: is it 404'ing, or stalling BTW? (just in case I'm mis-understanding!_) | 18:32 |
| timsim | Stalling, until all of the ' policy check succeeded for rule alltenants' go by, lol | 18:33 |
| timsim | With multiple workers, you can get the request (central processes - 1)/(central processes) % of the time | 18:33 |
| Kiall | And, the create_zone_import API call is returning before that? | 18:34 |
| timsim | It is, yes. | 18:35 |
| timsim | Which is odd, why is it that if I launch the tpool.execute in the main greenthread, it blocks that thread from continuing, but if i put it in another greenthread, it doesn't block the api from returning? | 18:36 |
| Kiall | Q - If you fire off an import, can you do a GET against say - /pools without it blocking? I'm kinda wondering if it's somehow related to you getting the import being processed somehow vs eventlet locking up? (Total guess, but worth a shot) | 18:39 |
| timsim | I wasn't able to get /v2/zones | 18:42 |
| Kiall | Oh well, was worth a try.. | 18:42 |
| Kiall | Firing up devstack with your patch + that pastebin ATM to give it a trey | 18:42 |
| Kiall | try | 18:42 |
| timsim | I appreciate it, especially since it's almost 8 pm :/ | 18:43 |
| Kiall | No worries.. Just wish devstack-from-scratch was faster ;) | 18:50 |
| timsim | This is the request I've been using. http://paste.openstack.org/show/221081/ | 18:52 |
| Kiall | So, 600 records? | 18:53 |
| timsim | Yeah, i was doing 200, but it was kind of hard to try things while the domain was being created, so I just pasted a bunch more on there. | 18:54 |
| Kiall | happen to have the CURL command handy? ;) | 18:56 |
| Kiall | cURL* ;) | 18:56 |
| timsim | curl -X POST localhost:9001/v2/zones/tasks/imports "Content-type:text/dns" --data-binary "@zone3.txt" -H Content-Type:text/dns | 18:57 |
| timsim | wait, wat, theres an unnecessary header bit in there | 18:57 |
| Kiall | no worries :) | 18:58 |
| *** jmcbride has joined #openstack-dns | 18:58 | |
| Kiall | Oh wow - that's logging lots | 18:59 |
| *** pglass has quit IRC | 19:00 | |
| Kiall | https://github.com/openstack/designate/blob/master/designate/central/service.py#L378 | 19:02 |
| *** jmcbride has quit IRC | 19:02 | |
| Kiall | ^ that can be cleaned up to cache the list of subdomains, saving us all sorts of extra SQL queries during an import | 19:02 |
| *** pfreund has quit IRC | 19:03 | |
| openstackgerrit | Graham Hayes proposed openstack/designate: Add Backend grading, and associated docs page. https://review.openstack.org/182414 | 19:03 |
| timsim | Being able to bulk input those recordsets would be life changing | 19:03 |
| mugsie | ++ | 19:04 |
| *** jmcbride has joined #openstack-dns | 19:04 | |
| Kiall | Yea, replacing that method with "return True" cleaned the logs up plenty | 19:05 |
| Kiall | still need a larger screen buffer tho, lol | 19:05 |
| *** pglass has joined #openstack-dns | 19:07 | |
| *** SunnyRainbow has joined #openstack-dns | 19:08 | |
| Kiall | timsim: .. so, the stuff offloaded to a read thread isn't actually taking mich time at all | 19:13 |
| Kiall | http://paste.openstack.org/show/221084/ | 19:13 |
| timsim | Yeah, I think the only time that would take a long time is if you had a huge zone file, I think I said before, the bulk of the time right now is being spent in create_domain. | 19:14 |
| *** openstackgerrit_ has joined #openstack-dns | 19:16 | |
| Kiall | timsim: Yea, there's like 4.5 million SQL queries happening while it's all locked up | 19:16 |
| Kiall | kinda looks like the same 3 or 4 queries over + over, hard to tell | 19:18 |
| *** vinod has quit IRC | 19:18 | |
| timsim | I think when you create domain w/ recordsets in it, it goes through and creates those recordsets 1 by 1, which in turn goes and creates the records 1 by 1. I'd assume that's what it is. | 19:18 |
| Kiall | Well, Yep - It does that, but I think we're doing lots in _is_valid_recordset_placement and _is_valid_recordset_placement_subdomain and .. etc | 19:19 |
| timsim | Ah yeah, makes sense | 19:20 |
| *** vinod has joined #openstack-dns | 19:24 | |
| *** openstackgerrit_ has quit IRC | 19:25 | |
| Kiall | timsim: so.. yea, the big issue is that we're doing lots of work in a single central call, where there's no yielding I/O points | 19:28 |
| Kiall | adding time.sleep(0) (which causes a yield, without adding an actual delay) here https://github.com/openstack/designate/blob/master/designate/central/service.py#L868 | 19:28 |
| Kiall | makes it not lock up | 19:28 |
| Kiall | http://paste.openstack.org/show/221098/ | 19:29 |
| Kiall | but.. still managed to get the waiter.switch() | 19:31 |
| Kiall | error: cannot switch to a different thread | 19:31 |
| Kiall | if I made a second request quick enough | 19:32 |
| *** shakamunyi has quit IRC | 19:35 | |
| *** barra204 has quit IRC | 19:36 | |
| timsim | Sorry, in a meeting. Any idea how to get around that threading business? | 19:43 |
| Kiall | Well, the bones of it is.. We're doing CRAPLOADS of work over and over rather than taking 1 pass at at.. | 19:44 |
| Kiall | (central, up until recently, has never needed to deal with more than "Do 1 thing and tell me when you're done" | 19:44 |
| Kiall | So... Batching up INSERTS, is an obvious one to get right.. But.. things like _is_valid_recordset_placement and _is_valid_recordset_placement_subdomain still need to happen, we should just do 1 pass over the lot, rather than looping and calling "find_domains" etc about 3 times per rrset and so on | 19:46 |
| timsim | Yeah. I was hoping to be able to that work in the background in a way that didn't impact other central operations, while knowing that later if we optimize that, it'll just be better. | 19:48 |
| Kiall | Anyway, for the purpose of bringing import back in.. I'd drop a feature flag on it (aka add a "enable_import_experimental" conf option ;)) and the time.sleep(0) into that loop .. then we can take a hammer to the create_domain() method to make it less stupid in the face of having to create some RRSets too | 19:48 |
| Kiall | Even if we move those half million SQL queries (Ok, probbaly an exaggeration - But it looked like 10k or so at least) into a separate OS thead, it's still going to cause issues with the DB | 19:50 |
| *** shakamunyi has joined #openstack-dns | 19:50 | |
| *** barra204 has joined #openstack-dns | 19:50 | |
| * Kiall checks how many SQL queries... | 19:52 | |
| timsim | That's a good point. | 19:52 |
| *** jmcbride has quit IRC | 19:56 | |
| openstackgerrit | Graham Hayes proposed openstack/designate: Add Backend grading, and associated docs page https://review.openstack.org/182414 | 19:57 |
| Kiall | timsim: http://paste.openstack.org/show/221117/ | 20:00 |
| *** jmcbride has joined #openstack-dns | 20:00 | |
| timsim | Alright, so as a temporary plan: bring in import with the sleep in create_domain w/ recordsets, as a next step, go hack on the create_domain process to make it better and then we can maybe remove that sleep(0). ? | 20:07 |
| Kiall | Yea, at the same time.. I'm trying to wrap my head around the ~100 SELECTs per 1 RRset happening there. | 20:08 |
| Kiall | I just can't see how that could even be remotely possible | 20:09 |
| timsim | Yeah that's odd | 20:10 |
| harmw | cliles: pign | 20:16 |
| harmw | ping | 20:16 |
| Kiall | timsim: yea, we're doing a pile of extra work... This is a tomorrow problem for me, too late in the day to try really dig into this.. | 20:24 |
| timsim | Yeah for sure, thanks for your help Kiall. I suppose the waiter.switch() problem is still an issue. | 20:28 |
| Kiall | Oh, right.. forgot about that one :/ | 20:28 |
| *** logan2 has quit IRC | 20:33 | |
| *** openstackgerrit has quit IRC | 20:37 | |
| *** openstackgerrit has joined #openstack-dns | 20:37 | |
| *** ChanServ sets mode: +v openstackgerrit | 20:37 | |
| timsim | Better just write a new service ;) | 20:40 |
| cliles | harmw: what is up? | 20:44 |
| *** vinod has quit IRC | 21:09 | |
| *** EricGonczer_ has quit IRC | 21:23 | |
| *** chlong has quit IRC | 21:24 | |
| *** SunnyRainbow has quit IRC | 21:30 | |
| *** jmcbride has quit IRC | 21:32 | |
| *** pglass has quit IRC | 21:35 | |
| *** andrewbogott is now known as andrewbogott_afk | 21:55 | |
| *** andrewbogott_afk is now known as andrewbogott | 22:11 | |
| *** logan2 has joined #openstack-dns | 22:18 | |
| *** chirag_ has quit IRC | 22:42 | |
| *** ducttape_ has quit IRC | 23:05 | |
| *** km has joined #openstack-dns | 23:06 | |
| *** SunnyRainbow has joined #openstack-dns | 23:12 | |
| *** chlong has joined #openstack-dns | 23:15 | |
| *** EricGonczer_ has joined #openstack-dns | 23:16 | |
| *** kei_yama has joined #openstack-dns | 23:23 | |
| *** SunnyRainbow has quit IRC | 23:38 | |
| *** EricGonc_ has joined #openstack-dns | 23:49 | |
| *** EricGonczer_ has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!