| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate-tempest-plugin master: Fixed multiple leaking tests https://review.opendev.org/c/openstack/designate-tempest-plugin/+/796375 | 06:03 |
|---|---|---|
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate-tempest-plugin master: Fixed multiple leaking tests https://review.opendev.org/c/openstack/designate-tempest-plugin/+/796375 | 07:35 |
| opendevreview | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Skip "test_list_all_projects_recordsets" because of Designate bug https://review.opendev.org/c/openstack/designate-tempest-plugin/+/796469 | 13:27 |
| *** njohnston_ is now known as njohnston | 14:22 | |
| opendevreview | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Fix "PTR recordset" tests suite https://review.opendev.org/c/openstack/designate-tempest-plugin/+/794708 | 14:27 |
| opendevreview | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Fix "PTR recordset" tests suite https://review.opendev.org/c/openstack/designate-tempest-plugin/+/794708 | 14:29 |
| jrosser | johnsom: should i see continuous polling from designate worker checking all the zones that exist? | 15:56 |
| johnsom | There is some polling to check the status yes. | 15:57 |
| jrosser | i'm just trying to debug a situation where a dns server got replaced, and newly created designate records are created OK, but old ones are not being put back | 15:59 |
| johnsom | jrosser I'm assuming the primary for the zone is designate? If so, check the tsig key is correct on the new DNS server. Also, see if you can force a zone transfer on the new DNS server for the zone. Otherwise it may wait for the SOA timeout. Which DNS server are you using? | 16:02 |
| jrosser | it's bind9 | 16:03 |
| johnsom | Actually, no, that doesn't really make sense. If it can do a transfer for new records, it should be transferring the whole zone. | 16:03 |
| jrosser | i was expecting to see a bunch of port 53 traffic to bind9 going round checking that the zones were all in place | 16:04 |
| jrosser | this sort of thing https://github.com/openstack/designate/blob/master/designate/worker/tasks/zone.py#L359 | 16:04 |
| jrosser | but it's quite possible thats not as continuous a process as i may be expecting | 16:04 |
| johnsom | https://docs.openstack.org/designate/latest/admin/config.html#producer-task-periodic-exists | 16:05 |
| johnsom | Producer triggers that, there is a timer setting there | 16:05 |
| jrosser | ahha | 16:06 |
| eandersson | johnsom: I still havent figured out a clean way to clean up zones created by these tests https://github.com/openstack/designate-tempest-plugin/blob/master/designate_tempest_plugin/tests/api/v2/test_zones_imports.py | 16:42 |
| eandersson | I kinda wish we had a utility or eve a test at the end that verified that the environment is clean. | 16:42 |
| mugsie | does "self.addCleanup()" not catch them? | 16:45 |
| jrosser | hmm so i see 'Finished emitting 48 events for shards 0 to 4094' in producer but really the log in the worker is really quiet | 16:45 |
| mugsie | jrosser: not all of the periodic tasks are on by default afaik | 16:46 |
| jrosser | mugsie: even though i see the events in the producer log? or at the worker end? | 16:47 |
| mugsie | so there is a couple of events that get triggered, the timer looks at what needs to run, then emits events for each of them | 16:48 |
| mugsie | the periodic exists event may not be included by default - I will have a look in a few mins | 16:48 |
| eandersson | mugsie yea the self.addCleanup code does not work at all for those tests | 16:49 |
| eandersson | or at least it does not remove the zones :D | 16:49 |
| johnsom | eandersson I was going to look at that today, but Tuesday seems to be the new Monday with all of the interruptions I am getting. | 16:50 |
| eandersson | No hurry. | 16:53 |
| mugsie | jrosser: oh, my bad - the "exists" task is for celiometer exists events | 16:55 |
| mugsie | it looks like there is a missing task :/ - we used to have a task that checked all customer facing DNS servers, and the DNS serial for them, and re-triggered a zone transfer if the zone was out of sync, but I can't see it anymore | 16:59 |
| johnsom | Yeah, I remember that. Producer used to trigger that. | 17:00 |
| mugsie | I think it might have been in the old zone manager days | 17:01 |
| mugsie | sorry, pool-manager | 17:01 |
| mugsie | timsim isn't here for me to abuse him about it | 17:03 |
| jrosser | mugsie: yeah thats the situation im in, having added another bind instance it's only getting new things added to it | 17:06 |
| mugsie | for bind, copying the zonefile + cache should be enough to pre-seed the new instance, but I think something the triggers at least a zone notify on a regular basis may be in order. | 17:10 |
| mugsie | and then for full completeness, a task that scans members of a pool, and marks the zones in error, which will get caught by the periodic recovery | 17:11 |
| mugsie | which will force a create | 17:11 |
| eandersson | https://github.com/openstack/designate/commit/3756fc51e71aaf0ba7cfb9155ca5d1de26ab78bc#diff-2187cbd3c9fec4b2af1d87e1d4d87c186219b2448f422e2e4ea338d765794dab | 17:14 |
| eandersson | I think this was added for this | 17:14 |
| eandersson | https://bugs.launchpad.net/charm-designate-bind/+bug/1879798 | 17:16 |
| johnsom | Joy. So, another option is to use rndc on the bind instance, assuming the zone exists, "rndc retransfer <zone>" | 17:17 |
| eandersson | but haven't read the full thread so might be completely off here :D | 17:17 |
| jrosser | eandersson: awesome thankyou - thats indeed triggered it to refresh the whole pool | 17:27 |
| jrosser | i guess i was assuming too much about how proactively designate could deal with a new/re-provisioned dns server | 17:28 |
| johnsom | I made a note of this for our documentation update tasks. | 17:39 |
| opendevreview | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Fix "PTR recordset" tests suite https://review.opendev.org/c/openstack/designate-tempest-plugin/+/794708 | 17:41 |
| eandersson | johnsom looking good! 2 last runs were successful. I am feeling lucky with the third one. | 18:05 |
| eandersson | as long as the rng does not hit the tld bug it should succeed :D | 18:05 |
| johnsom | lol | 18:05 |
| johnsom | If only we could mock out the rng.... | 18:06 |
| eandersson | I mean in theory we could just create a tld for ptrs when running the ptrs test, but ideal would be if we could just run the tld tests separeate? | 18:06 |
| johnsom | Yeah, we need to step back and look at that. Since any TLD flips the bit for all zone creates, we need to be careful how we handle them in the test suite. | 18:07 |
| eandersson | At least in my local testing I hit it about maybe <5% of the time now. So it hopefully gives us some breathing room | 18:10 |
| eandersson | to come up with a real fix | 18:10 |
| johnsom | It's looking good. I'm reviewing now. I think we can land this patch, rebase arkady's on it and go from there. | 20:29 |
| johnsom | There is overlap between the two, but this approach should just keep things simple | 20:29 |
| eandersson | Yea - I kept it as simple as possible. More things I want to fix, but that can be later. Ideally you should be able to run tempest multiple times in a row, but that still fails. | 20:31 |
| eandersson | (without having to manually clean up) | 20:32 |
| johnsom | Yeah, agreed. It really should be "safe" to run on a production cloud. In theory. | 20:35 |
| eandersson | Feel free to merge it in johnsom | 21:00 |
| eandersson | nicolasbock +2d it already | 21:00 |
| johnsom | lol, two more files to look at, then | 21:00 |
| eandersson | there are some unrelated changes I missed (like new lines) | 21:01 |
| eandersson | but personally dont really care :D | 21:01 |
| johnsom | Yeah, I'll give you a pass on those in this case. girn | 21:01 |
| johnsom | I would have been done, but just got another surprise email/invite... sigh | 21:03 |
| nicolasbock | I thought it was ok to lump those beautifications into the change eandersson :) | 21:07 |
| johnsom | He knows I would nit him usually. lol | 21:08 |
| johnsom | Little merge conflict magnets | 21:09 |
| johnsom | I'll rebase the other patch now | 21:12 |
| opendevreview | Michael Johnson proposed openstack/designate-tempest-plugin master: Fix "PTR recordset" tests suite https://review.opendev.org/c/openstack/designate-tempest-plugin/+/794708 | 21:28 |
| eandersson | btw something that would be nice would be to include the test (or parts of the test) in the random zone names. | 21:41 |
| johnsom | Yes, that should always be the case | 21:41 |
| eandersson | It's such a pain to hunt down what created a zone at the moment with the completely randomized names. | 21:41 |
| eandersson | I kinda wish we allowed comments / descriptions for zones | 21:42 |
| johnsom | So, looking at this zone import issue you mentioned, do you have a pointer to the test that is leaving things around? I need to restack to run this local. | 21:42 |
| eandersson | I have just been running this over and over | 21:54 |
| eandersson | > designate_tempest_plugin.tests.api.v2.test_zones_imports.ZonesImportTest.test_create_zone_import | 21:54 |
| eandersson | I don't fully understand the zone import code, but neither show or list zone import contains the uuid in any of the zones created | 21:56 |
| johnsom | Ha, yeah, ok I see that the zone create throws away the details of the zone created. My pet-peeve "_" dummy variable used there. | 21:59 |
| johnsom | We need to capture that response, parse it for the zone ID, then register that for cleanup | 22:00 |
| johnsom | Probably a async mess there too from the looks of it. | 22:03 |
| opendevreview | Merged openstack/designate-tempest-plugin master: Fixed multiple leaking tests https://review.opendev.org/c/openstack/designate-tempest-plugin/+/796375 | 22:19 |
| eandersson | Another idea would be to just capture the zone name and just find it and delete it using the name instead. | 23:07 |
| eandersson | btw there is also one PTR record that I can't figure out where it is created | 23:08 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!