| *** EricGonczer_ has joined #openstack-dns | 00:05 | |
| *** james_li_ has quit IRC | 00:08 | |
| *** rsyed is now known as rsyed_away | 00:21 | |
| *** rsyed_away is now known as rsyed | 00:25 | |
| *** rsyed is now known as rsyed_away | 00:47 | |
| *** ccneill has quit IRC | 00:57 | |
| *** penick has quit IRC | 00:57 | |
| *** EricGonczer_ has quit IRC | 01:01 | |
| *** ducttape_ has joined #openstack-dns | 01:01 | |
| *** chlong has joined #openstack-dns | 01:01 | |
| *** rudrajit has quit IRC | 01:10 | |
| *** ducttape_ has quit IRC | 01:24 | |
| *** chlong has quit IRC | 01:34 | |
| *** chlong has joined #openstack-dns | 01:35 | |
| *** raginbajin has quit IRC | 01:42 | |
| *** raginbajin has joined #openstack-dns | 01:46 | |
| *** ducttape_ has joined #openstack-dns | 02:03 | |
| *** jasonsb has joined #openstack-dns | 02:10 | |
| *** jasonsb has quit IRC | 02:12 | |
| *** jasonsb has joined #openstack-dns | 02:13 | |
| *** rudrajit has joined #openstack-dns | 02:25 | |
| *** EricGonczer_ has joined #openstack-dns | 02:27 | |
| *** EricGonczer_ has quit IRC | 02:47 | |
| *** johnbelamaric has joined #openstack-dns | 02:49 | |
| *** johnbelamaric has quit IRC | 02:59 | |
| *** EricGonczer_ has joined #openstack-dns | 03:16 | |
| *** ducttape_ has quit IRC | 03:35 | |
| *** ducttape_ has joined #openstack-dns | 03:37 | |
| *** EricGonczer_ has quit IRC | 03:49 | |
| *** km has quit IRC | 04:03 | |
| *** km has joined #openstack-dns | 04:03 | |
| *** ducttape_ has quit IRC | 04:08 | |
| *** mlavalle has quit IRC | 04:43 | |
| *** km__ has joined #openstack-dns | 04:55 | |
| *** km has quit IRC | 04:55 | |
| *** km__ is now known as Guest24021 | 04:56 | |
| *** rudrajit has quit IRC | 04:56 | |
| *** rudrajit has joined #openstack-dns | 05:11 | |
| *** GonZo2K has quit IRC | 05:26 | |
| *** rudrajit has quit IRC | 06:52 | |
| *** nyechiel_ has joined #openstack-dns | 06:53 | |
| *** venkat has joined #openstack-dns | 07:03 | |
| *** venkat has quit IRC | 07:16 | |
| *** chlong has quit IRC | 08:52 | |
| *** karimb has joined #openstack-dns | 09:01 | |
| *** kei_yama has quit IRC | 09:04 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-dns | 09:17 | |
| *** ChanServ sets mode: +v openstackgerrit | 09:17 | |
| *** jordanP has joined #openstack-dns | 09:23 | |
| *** boris-42 has joined #openstack-dns | 09:52 | |
| *** richm has joined #openstack-dns | 10:05 | |
| *** GonZo2K has joined #openstack-dns | 10:18 | |
| *** GonZo2K has quit IRC | 10:31 | |
| *** karimb has quit IRC | 10:31 | |
| *** karimb_ has joined #openstack-dns | 10:31 | |
| *** karimb_ has quit IRC | 10:36 | |
| *** GonZo2K has joined #openstack-dns | 10:41 | |
| *** karimb has joined #openstack-dns | 10:47 | |
| *** nkinder has joined #openstack-dns | 11:00 | |
| *** karimb has quit IRC | 11:08 | |
| *** karimb has joined #openstack-dns | 11:09 | |
| *** sonuk has quit IRC | 11:12 | |
| *** sonuk has joined #openstack-dns | 11:12 | |
| *** karimb has quit IRC | 11:13 | |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Minor test cleanup https://review.openstack.org/266322 | 11:26 |
|---|---|---|
| *** GonZo2K has quit IRC | 11:40 | |
| *** bauruine has quit IRC | 11:51 | |
| *** bauruine has joined #openstack-dns | 11:57 | |
| *** nkinder has quit IRC | 11:59 | |
| *** nyechiel_ has quit IRC | 12:04 | |
| *** en_austin has joined #openstack-dns | 12:15 | |
| en_austin | Kiall federico3 ping? | 12:16 |
| federico3 | yep | 12:16 |
| en_austin | just to remind you: last Friday we with you and Kiall investigated a PM halting | 12:46 |
| en_austin | do you remember? :) | 12:46 |
| en_austin | so I'm here to provide a short report | 12:47 |
| en_austin | in general - seems that removing of "with lockutils.lock" line from PM solved my problem - and it's running well as for now | 12:48 |
| en_austin | so thank you Kiall & federico3 for help :) | 12:48 |
| en_austin | but I have another question :) today I've encountered such situation - 8 mDNS workers, ~5-10 requests per 10 seconds, and 2/8 mDNS's are eating 100% CPU (as htop says), another 6 are idle (2-3%) | 12:49 |
| en_austin | and users are reporting a degradated records propagation (from 20-30s to 1-2min) | 12:50 |
| en_austin | are there a way to increase a performance at this part? :) | 12:50 |
| *** Guest24021 has quit IRC | 12:51 | |
| en_austin | also, I've found a logs with "unexpected soa from backend" (I think because of PM process w/o lock and >1 mDNS process, but it's only my guess | 12:51 |
| federico3 | en_austin: removing the locking prevented the deadlocking of the greenthreads that you were seeing on the debugging console? | 12:54 |
| en_austin | Yeah, exactly. | 12:54 |
| federico3 | en_austin: the one in update_status(), right? | 12:55 |
| *** ducttape_ has joined #openstack-dns | 12:58 | |
| federico3 | uhm, unexpected soa? do you have that log line handy? | 12:59 |
| *** EricGonczer_ has joined #openstack-dns | 13:06 | |
| openstackgerrit | Endre Karlson proposed openstack/designate: Move config to avoid import issues with eventlet https://review.openstack.org/236987 | 13:07 |
| openstackgerrit | Endre Karlson proposed openstack/designate: Support running Designate API in Apache 2 https://review.openstack.org/236236 | 13:07 |
| *** ducttape_ has quit IRC | 13:16 | |
| *** boris-42 has quit IRC | 13:23 | |
| *** nkinder has joined #openstack-dns | 13:50 | |
| en_austin | 2016-02-05 16:50:58.630 6897 WARNING designate.mdns.notify [req-c05860b2-e589-4e90-89f9-d6923109f68e noauth-user noauth-project - - -] Got lower serial for 'aqa.int.zone.' to '10.28.0.18:53'. Expected:'1454680253'. Got:'1454680218'.Retries left='9' | 13:51 |
| en_austin | federico3 | 13:51 |
| en_austin | this appears when >1 record is created in the moment (with 1-2 sec delay) | 13:51 |
| en_austin | seems is that lock in PM was there to prevent such things) | 13:52 |
| federico3 | oh that "lower serial" is expected in this case... as long as later on the propagation succeeds | 13:55 |
| *** sonuk has quit IRC | 14:19 | |
| *** karimb has joined #openstack-dns | 14:20 | |
| *** rsyed_away is now known as rsyed | 14:24 | |
| en_austin | awh, okay | 14:24 |
| en_austin | any thoughts abt 100% cpu load? | 14:25 |
| *** karimb has quit IRC | 14:25 | |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Switch "managed" fields from unicode to string https://review.openstack.org/276448 | 14:30 |
| *** jordanP has quit IRC | 14:56 | |
| *** ducttape_ has joined #openstack-dns | 14:58 | |
| *** mlavalle has joined #openstack-dns | 15:18 | |
| *** nkinder has quit IRC | 15:18 | |
| *** richm has quit IRC | 15:26 | |
| *** nkinder has joined #openstack-dns | 15:28 | |
| *** pglass has joined #openstack-dns | 15:39 | |
| *** logan_ has joined #openstack-dns | 16:00 | |
| *** logan_ is now known as Guest78654 | 16:01 | |
| *** logan- has quit IRC | 16:01 | |
| *** Guest78654 is now known as logan- | 16:02 | |
| *** default has joined #openstack-dns | 16:04 | |
| *** default is now known as Guest77203 | 16:04 | |
| *** default_ has quit IRC | 16:14 | |
| *** nkinder has quit IRC | 16:14 | |
| *** EricGonc_ has joined #openstack-dns | 16:21 | |
| *** EricGonczer_ has quit IRC | 16:24 | |
| *** ducttape_ has quit IRC | 16:25 | |
| *** ducttape_ has joined #openstack-dns | 16:49 | |
| *** bpokorny has joined #openstack-dns | 16:51 | |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Switch "managed" fields from unicode to string https://review.openstack.org/276448 | 16:55 |
| *** nkinder has joined #openstack-dns | 17:05 | |
| *** richm has joined #openstack-dns | 17:06 | |
| *** en_austin has quit IRC | 17:18 | |
| *** nkinder has quit IRC | 17:27 | |
| *** en_austin has joined #openstack-dns | 17:28 | |
| federico3 | en_austin: is it constantly 100% on the same hosts? Did you take a look through the backdoor, is there any greenlet stuck or are they all doing work? | 17:37 |
| en_austin | Awh, I've forgot to dump a backdoor data :( | 17:38 |
| en_austin | What do you mean under "100% on the same hosts"? | 17:38 |
| federico3 | are they at 100% forever or for some time only? | 17:38 |
| en_austin | some time, surely | 17:38 |
| federico3 | I mean: is it the same 2 hosts doing 100% all the time or sometime it's some other hosts getting all the load? | 17:38 |
| en_austin | which "hosts" you are talking about? I'm running an single Designate instance at the same virtual environment (OpenVZ container) | 17:39 |
| federico3 | "2/8 mDNS's are eating 100% CPU" <- ok, this are workers on the same host but different containers then | 17:40 |
| federico3 | ...with 10 different ipaddrs I suppose? | 17:40 |
| en_austin | Hmmm.... nope | 17:40 |
| en_austin | it's only workers=8 in mDNS section config | 17:40 |
| en_austin | there was only workers=1 before (and consuming 100% cpu, so I've decided to increase this count to 8) | 17:41 |
| en_austin | http://paste.openstack.org/show/dxFyS4Ba8Z9E9HR30uWR/ here is my Designate config (but with "8" value at line 92, as for now) | 17:42 |
| *** karimb has joined #openstack-dns | 17:47 | |
| *** karimb has quit IRC | 17:49 | |
| en_austin | federico3: i'm running all D services on the same box, using 127.0.0.1 for communication between them | 17:54 |
| *** richm has quit IRC | 17:54 | |
| *** richm has joined #openstack-dns | 17:55 | |
| *** penick has joined #openstack-dns | 17:56 | |
| federico3 | it could be due to the sharding: are you updating many tenths/hundreds of different zones or just a few? | 18:00 |
| eandersson | out of interest which version of Designate is this en_austin ? | 18:07 |
| en_austin | stable/liberty @ 94d0f00b09044fcef36604a9c81accb0907c9705 | 18:07 |
| eandersson | thanks - sounds like I should be holding off upgrading then | 18:08 |
| en_austin | federico3: i'm running only two zones | 18:08 |
| *** richm has quit IRC | 18:08 | |
| *** rudrajit has joined #openstack-dns | 18:08 | |
| en_austin | eandersson: i've upgraded from Kilo since I've encountered several unrecoverable issues with records propagating (e.g it halted), and upgrade solved this | 18:08 |
| *** rudrajit has quit IRC | 18:09 | |
| eandersson | hmm havent had that in Kilo. The only issue I had was with mdns not closing tcp connections properly, but Kiall released a patch for that | 18:09 |
| *** rudrajit_ has joined #openstack-dns | 18:10 | |
| en_austin | eandersson have you ever saw a situation when PoolManager suddenly stops to propagate records (or was doing it extremely slowly)? | 18:10 |
| eandersson | Yep | 18:11 |
| eandersson | That was the issue | 18:11 |
| en_austin | (I've encountered this couple of weeks ago, and found a solution) | 18:11 |
| en_austin | federico3 & Kiall helped me | 18:11 |
| en_austin | do you still need to solve it? | 18:11 |
| eandersson | The odd thing is that it was solved by fixing the TCP issue in mdns | 18:11 |
| eandersson | Or at least I haven't seen it since | 18:11 |
| en_austin | well | 18:11 |
| en_austin | we've found a deadlock in PM greenlets processes | 18:12 |
| eandersson | Oh, that affects Kilo as well? | 18:12 |
| eandersson | Is there a patch in the review queue? | 18:12 |
| en_austin | I don't know about a patch, but I can share a patch that Kiall gave to me a week ago with comments "let's test it" | 18:13 |
| eandersson | Sure | 18:13 |
| eandersson | That would be great | 18:13 |
| en_austin | since they cannot reproduce it on theis instances, I supposed to be a beta tester :) | 18:13 |
| en_austin | and, to tell the truth, it helped | 18:13 |
| en_austin | http://eavesdrop.openstack.org/irclogs/%23openstack-dns/%23openstack-dns.2016-01-29.log.html | 18:14 |
| eandersson | I'll test it if I run into this issue again. | 18:14 |
| en_austin | see msg from Kiall from 17:27 | 18:14 |
| en_austin | (and you can watch the whole conversation, maybe - it's your problem too) | 18:14 |
| eandersson | I haven't seen it since I applied https://github.com/openstack/designate/commit/a170caa870dbb7ae90e5df6ad532d968f9e15b86 | 18:15 |
| eandersson | but I will test it if it happens again | 18:15 |
| en_austin | How many zones/records do you have at your D instance and what is your query rate to API (e.g CRUDs/minute, for example) | 18:18 |
| en_austin | just to compare with mine | 18:19 |
| eandersson | query rate to api is low | 18:20 |
| *** ducttape_ has quit IRC | 18:21 | |
| en_austin | i've encountered a deadlock in PM with query rate ~50 req/min | 18:21 |
| eandersson | This is for a private cloud, so we might not be encoutering it due to that | 18:22 |
| eandersson | We are upgrading to Kilo in our dev environment so might hit it then | 18:22 |
| eandersson | Since that one will have significnatly more vms | 18:22 |
| en_austin | I'm also running private dev cloud in our company :) | 18:22 |
| en_austin | You're running Juno in your production? | 18:23 |
| eandersson | Kilo in prod | 18:23 |
| eandersson | but dev is behind | 18:23 |
| eandersson | Dev Designate was running Juno :D | 18:24 |
| eandersson | Designate/Juno was a terrible experience haha spent so many hours working on getting that stable | 18:28 |
| *** pglass has quit IRC | 18:33 | |
| *** ducttape_ has joined #openstack-dns | 18:34 | |
| en_austin | :D | 18:38 |
| *** rsyed is now known as rsyed_away | 18:39 | |
| *** en_austin has quit IRC | 18:41 | |
| *** rsyed_away is now known as rsyed | 18:42 | |
| *** richm has joined #openstack-dns | 18:50 | |
| *** rsyed is now known as rsyed_away | 19:02 | |
| *** pglass has joined #openstack-dns | 19:06 | |
| *** ducttape_ has quit IRC | 19:21 | |
| *** ducttape_ has joined #openstack-dns | 19:36 | |
| *** richm has quit IRC | 19:46 | |
| *** rudrajit_ has quit IRC | 19:52 | |
| *** penick has quit IRC | 20:25 | |
| openstackgerrit | Paul Glass proposed openstack/designate: A request hook interface for the functional test client https://review.openstack.org/276905 | 20:26 |
| *** rsyed_away is now known as rsyed | 20:33 | |
| openstackgerrit | Merged openstack/designate: Log stacktraces of MultipleExceptions in functionaltests https://review.openstack.org/274807 | 20:51 |
| *** rudrajit has joined #openstack-dns | 21:10 | |
| *** rudrajit has quit IRC | 21:14 | |
| *** boris-42 has joined #openstack-dns | 21:23 | |
| *** sonuk has joined #openstack-dns | 21:30 | |
| *** pglbutt has joined #openstack-dns | 21:32 | |
| *** pglass_ has joined #openstack-dns | 21:33 | |
| *** pglass has quit IRC | 21:34 | |
| *** rudrajit has joined #openstack-dns | 21:34 | |
| *** pglbutt has quit IRC | 21:37 | |
| *** en_austin has joined #openstack-dns | 21:40 | |
| *** en_austin has quit IRC | 21:43 | |
| *** en_austin has joined #openstack-dns | 21:44 | |
| *** richm has joined #openstack-dns | 22:17 | |
| *** GonZo2K has joined #openstack-dns | 22:19 | |
| *** en_austin has quit IRC | 22:28 | |
| *** rsyed is now known as rsyed_away | 22:31 | |
| *** en_austin has joined #openstack-dns | 22:38 | |
| *** ducttape_ has quit IRC | 22:41 | |
| openstackgerrit | Paul Glass proposed openstack/designate: WIP: A request hook interface for the functional test client https://review.openstack.org/276905 | 22:55 |
| *** en_austin has quit IRC | 22:56 | |
| *** richm has quit IRC | 22:56 | |
| *** mlavalle has quit IRC | 23:12 | |
| *** pglass_ has quit IRC | 23:23 | |
| *** chlong has joined #openstack-dns | 23:38 | |
| *** mlavalle has joined #openstack-dns | 23:49 | |
| *** ducttape_ has joined #openstack-dns | 23:58 | |
| *** GonZo2K has quit IRC | 23:58 | |
| *** mlavalle has quit IRC | 23:59 | |
| *** GonZo2K has joined #openstack-dns | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!