| *** trungnv has quit IRC | 01:16 | |
| *** trungnv has joined #openstack-dns | 01:17 | |
| *** ivve has joined #openstack-dns | 02:09 | |
| *** marst has quit IRC | 04:02 | |
| *** marst has joined #openstack-dns | 04:08 | |
| *** marst has quit IRC | 04:19 | |
| *** Trieu-Van has joined #openstack-dns | 04:30 | |
| *** pcaruana has joined #openstack-dns | 04:31 | |
| *** Trieu-Van has quit IRC | 04:32 | |
| *** linhnm has joined #openstack-dns | 04:32 | |
| *** marst has joined #openstack-dns | 04:36 | |
| *** linhnm has quit IRC | 04:37 | |
| *** linhnm has joined #openstack-dns | 04:38 | |
| *** marst has quit IRC | 04:41 | |
| *** linhnm has quit IRC | 05:04 | |
| *** mikal has joined #openstack-dns | 05:12 | |
| *** mikal_ has quit IRC | 05:15 | |
| *** marst has joined #openstack-dns | 05:16 | |
| *** linhnm has joined #openstack-dns | 05:26 | |
| *** masber has joined #openstack-dns | 05:51 | |
| *** linhnm has quit IRC | 05:56 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/designate master: Add unqiue constraint to service_statuses https://review.openstack.org/568372 | 05:58 |
|---|---|---|
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/designate master: Add unqiue constraint to service_statuses https://review.openstack.org/568372 | 05:58 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/designate master: Add unqiue constraint to service_statuses https://review.openstack.org/568372 | 06:00 |
| eandersson | mugsie, frickler unsure about best practices for schema upgrades ^ | 06:00 |
| *** linhnm has joined #openstack-dns | 06:22 | |
| *** trungnv has quit IRC | 06:54 | |
| *** trungnv has joined #openstack-dns | 06:55 | |
| *** Tbone has joined #openstack-dns | 07:10 | |
| *** pcaruana has quit IRC | 07:17 | |
| *** pcaruana has joined #openstack-dns | 07:17 | |
| *** pcaruana has quit IRC | 07:18 | |
| *** pcaruana has joined #openstack-dns | 07:23 | |
| *** linhnm has quit IRC | 07:36 | |
| *** AlexeyAbashkin has joined #openstack-dns | 07:36 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/designate stable/queens: Update auth_uri option to www_authenticate_uri https://review.openstack.org/568499 | 07:58 |
| *** livelace has joined #openstack-dns | 07:59 | |
| *** salmankhan has joined #openstack-dns | 08:23 | |
| *** salmankhan has quit IRC | 08:59 | |
| *** salmankhan has joined #openstack-dns | 09:06 | |
| *** linhnm has joined #openstack-dns | 09:11 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/designate master: Allow to check pep8 along with D000 https://review.openstack.org/567458 | 09:19 |
| *** linhnm has quit IRC | 09:42 | |
| *** marst has quit IRC | 10:56 | |
| *** Tbone has quit IRC | 11:03 | |
| *** marst has joined #openstack-dns | 11:09 | |
| *** marst has quit IRC | 11:14 | |
| *** rfreire has joined #openstack-dns | 11:36 | |
| *** marst has joined #openstack-dns | 12:01 | |
| openstackgerrit | Merged openstack/designate master: Change copyright information for field.py https://review.openstack.org/565237 | 12:04 |
| *** rfreire has quit IRC | 13:34 | |
| *** rpittau has joined #openstack-dns | 13:40 | |
| *** rfreire has joined #openstack-dns | 14:00 | |
| *** rfreire has quit IRC | 14:02 | |
| *** rfreire has joined #openstack-dns | 14:02 | |
| *** waynr has joined #openstack-dns | 14:08 | |
| *** rfreire_ has joined #openstack-dns | 14:32 | |
| *** rfreire_ has quit IRC | 14:32 | |
| *** rfreire has quit IRC | 14:36 | |
| waynr | howdy folks | 14:38 |
| mugsie | o/ | 14:39 |
| waynr | so my team is currently dealing with a designate problem in which we are seeing designate repeatedly attempt to delete zones and getting 'rndc: delzone failed: notfound' | 14:41 |
| waynr | does that sound familiar to anyone? | 14:41 |
| waynr | we suspect the reason this initially started happening was that one of our nameservers went down the other day and designate's database somehow got out of sync with the nameserver state | 14:43 |
| mugsie | waynr: is it the same zone? | 14:47 |
| waynr | yeah it appears to be looping through the same set of zones repeatedly | 14:48 |
| mugsie | can you grab logs? | 14:50 |
| *** renmak_ has joined #openstack-dns | 15:06 | |
| waynr | this is what the error messages look like in the logs: http://paste.openstack.org/show/O7lQRk5Do9iLaYvF4bh2/ | 15:06 |
| *** renmak__ has joined #openstack-dns | 15:06 | |
| mugsie | is there any surrounding context? | 15:07 |
| waynr | if more context would be helpful i can get it, i'm just not the person logged in to machines debugging right now | 15:07 |
| mugsie | Yeah, I want to see what task is running to try and delete the zone | 15:08 |
| mugsie | I am assuming these zones were deleted during the bind server outage? | 15:08 |
| leader716 | yes it does seem that way | 15:09 |
| leader716 | it seems that the delete is stuck in a loop, it has been deleted successfully from both of the bind servers, but designate keeps trying to delete the zone | 15:10 |
| leader716 | which return the error in the command | 15:10 |
| waynr | the point person looking at the designate database says that the zones are marked as deleted in the database | 15:11 |
| waynr | (i'm still waiting to get the additional log context by the way) | 15:12 |
| leader716 | and it is marked as DELETED in the database with a deleted value of something other than 0 | 15:12 |
| mugsie | what layout are you using? Pool Manager or Worker? | 15:14 |
| leader716 | worker | 15:14 |
| mugsie | Oh. | 15:15 |
| leader716 | we did just notice that the rabbit worker queue is at 58173 | 15:15 |
| mugsie | Oh, it could have been a back log then | 15:15 |
| mugsie | how many zones / workers ? | 15:16 |
| leader716 | 3 workers, right now 4 zones are active | 15:16 |
| leader716 | let me double check on the zones | 15:17 |
| leader716 | 47 zones are active | 15:17 |
| leader716 | was only looking at my tenant initially | 15:18 |
| mugsie | that is a high amount of rmq messages to be built up | 15:19 |
| mugsie | A potential fix is, clear RMQ queue (at this point the messages are irelevent) | 15:22 |
| mugsie | then bounce workers | 15:23 |
| mugsie | this should not impact API calls at all | 15:23 |
| *** sapd1 has joined #openstack-dns | 15:45 | |
| *** salmankhan has quit IRC | 16:01 | |
| leader716 | @mugsio that seems to have resolved the issue | 16:01 |
| leader716 | mugsie: thanks for the help | 16:02 |
| mugsie | leader716: no problem | 16:02 |
| *** renmak_ has quit IRC | 16:04 | |
| *** renmak__ has quit IRC | 16:04 | |
| *** pcaruana has quit IRC | 16:33 | |
| *** AlexeyAbashkin has quit IRC | 17:25 | |
| *** cliles has quit IRC | 17:31 | |
| *** cliles has joined #openstack-dns | 17:36 | |
| *** rfreire has joined #openstack-dns | 17:47 | |
| *** salmankhan has joined #openstack-dns | 18:02 | |
| *** sapd1 has quit IRC | 18:22 | |
| *** KeithMnemonic has joined #openstack-dns | 19:14 | |
| *** KeithMnemonic1 has joined #openstack-dns | 19:14 | |
| *** KeithMnemonic1 has quit IRC | 19:15 | |
| *** salmankhan has quit IRC | 19:39 | |
| *** rfreire has quit IRC | 20:23 | |
| *** rfreire has joined #openstack-dns | 20:37 | |
| *** rfreire has quit IRC | 21:29 | |
| *** livelace has quit IRC | 22:18 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!