| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Cleaned up and fixed record objects and tests https://review.opendev.org/c/openstack/designate/+/849831 | 01:23 |
|---|---|---|
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Cleaned up and fixed record objects and tests https://review.opendev.org/c/openstack/designate/+/849831 | 01:24 |
| opendevreview | Merged openstack/designate master: Cleaned up adapter code https://review.opendev.org/c/openstack/designate/+/849842 | 02:19 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Cleaned up and fixed record objects and tests https://review.opendev.org/c/openstack/designate/+/849831 | 02:58 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Add basic information to represent DesignateObjects https://review.opendev.org/c/openstack/designate/+/849847 | 04:20 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Added additional test coverage for adapters https://review.opendev.org/c/openstack/designate/+/849848 | 04:33 |
| *** tkajinam is now known as Guest5435 | 07:14 | |
| *** Guest5435 is now known as tkajinam | 07:36 | |
| opendevreview | Andre Aranha proposed openstack/designate stable/wallaby: Change FIPS jobs to centos-9-stream https://review.opendev.org/c/openstack/designate/+/850333 | 11:39 |
| ozzzo_work | eandersson: Here's what I see in designate-api.log: https://paste.openstack.org/show/bSVwBcGWuQGGhrilEOCT/ | 14:15 |
| ozzzo_work | I don't see any new entries in designate-central.log when I reproduce the issue | 14:16 |
| ozzzo_work | In designate-sink.log I see lots of "NameError: name 'ptr_recordset' is not defined" and "designate.exceptions.DuplicateRecord: Duplicate Record" | 14:23 |
| ozzzo_work | Example: https://paste.openstack.org/show/brCAoLhDPsPdY9LG2mIv/ | 14:28 |
| ozzzo_work | I'm not sure that this is related to the other symptom, but the initial complaint about this region was that DNS entries were not created for new VMs. When I investigated then i saw the "recordset list" failure | 14:30 |
| ozzzo | We found that a customer who has tenant networking enabled has huge numbers of entries in the Designate database although they have no active VMs. It appears that, when they delete VMs, the corresponding Designate database entries do not get deleted. This causes the original symptom that the customer reported. I suspect that this deletion failure caused the Designate database to swell to the point where it causes "recordset | 14:43 |
| ozzzo | list" failures. | 14:43 |
| ozzzo | How does tenant networking interact with Designate? | 14:44 |
| ozzzo | Actually I'm not sure that "huge" is accurate; there are 32K entries in the DB and 16K of them belong to this customer | 14:44 |
| ozzzo | our non-broken region has 3K entries in the same DB | 14:45 |
| johnsom | ozzzo That is handled by the neutron DNS extensions. They call the Designate API (in newer versions) or in older versions it puts messages on the messaging bus for Designate. | 15:01 |
| ozzzo | we're on Train; is that using the API? | 15:02 |
| johnsom | I would have to dig in the neutron release notes to see when that switched. However, I did see this that might be related to your issue if you ran train older than 15.2.0: https://bugs.launchpad.net/neutron/+bug/1875981 | 15:28 |
| johnsom | https://docs.openstack.org/releasenotes/neutron/train.html#relnotes-15-2-0-stable-train | 15:28 |
| opendevreview | Merged openstack/designate master: Remove logic for Python < 3.2 https://review.opendev.org/c/openstack/designate/+/850092 | 16:40 |
| eandersson | ozzzo: If you are using the older, sink version it's possible that the notification queue stopped working | 17:50 |
| eandersson | designate-sink | 17:50 |
| eandersson | What orchestration are you using? e.g. kolla? | 17:51 |
| ozzzo | yes | 17:51 |
| ozzzo | we're using redhat's RHOSP 16.2 | 17:51 |
| eandersson | Interesting error btw. That looks to be something between haproxy and designate | 17:52 |
| eandersson | Maybe the call is taking too long to return and you have set haproxy to not wait long enough | 17:52 |
| eandersson | Yep that is it | 17:52 |
| eandersson | > time: 61.7501233 | 17:52 |
| eandersson | Your haproxy is configured to kill the TCP connection after 60 seconds | 17:52 |
| eandersson | https://stackoverflow.com/questions/32634980/haproxy-closes-long-living-tcp-connections-ignoring-tcp-keepalive | 17:52 |
| eandersson | I see that you figured out the cause already :D | 17:55 |
| ozzzo | I think so. We're going to identify and delete the bad entries in the Designate DB and see if that fixes it, then we just have to figure out (and fix) why tenant-networked VM entries are failing to delete when the VM is deleted | 17:57 |
| ozzzo | our other region with tenant networking enabled is less used; it has the same problem but hasn't escalated to the point where "recordset list" fails yet | 17:58 |
| opendevreview | Manish Honap proposed openstack/designate master: Validate matching double quotes in TXT recordsets. https://review.opendev.org/c/openstack/designate/+/849541 | 18:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!