| frickler | eandersson: I'm stuck debugging designate testing failures in kolla, maybe you can take a look https://1f951496931df7f98421-2c4ddb9b866dbe784d60fff20a0119fe.ssl.cf1.rackcdn.com/892278/2/check/kolla-ansible-ubuntu-magnum/ab9f889/primary/logs/ansible/test-magnum | 11:45 |
|---|---|---|
| frickler | a zone is created and then designate-sink is configured to create records in that zone | 11:46 |
| frickler | after restarting designate, the zone is stuck in PENDING/UPDATE, before any real update is made, while records in the zone are stuck in PENDING/CREATE | 11:46 |
| frickler | I have a node held for that job in case you'd want to take a look at things in situ | 11:47 |
| frickler | that job is failing since about 2 months, was working fine earlier, and I don't think anything has changed on the kolla side | 11:47 |
| frickler | the only patch merged in designate that I've found is https://review.opendev.org/c/openstack/designate/+/886621 but I fail to see how that could break things | 11:50 |
| frickler | this was more text about the issue than I expected, guess I should put it onto LP ;) | 11:51 |
| eandersson | Interesting. I'll take a look. | 13:59 |
| eandersson | frickler: Can you enable debug logs for designate? | 14:10 |
| frickler | eandersson: https://1f951496931df7f98421-2c4ddb9b866dbe784d60fff20a0119fe.ssl.cf1.rackcdn.com/892278/2/check/kolla-ansible-ubuntu-magnum/ab9f889/primary/logs/kolla/designate/index.html should have them | 14:12 |
| eandersson | Yea just found them was just looking in the wrong place | 14:13 |
| frickler | yes, kolla job logs are a bit difficult to look at if you're not used to it, sorry for that | 14:14 |
| eandersson | The producer isn't doing anything which is odd | 14:15 |
| eandersson | https://zuul.opendev.org/t/openstack/build/b3a983600d6347a3abaa819461986d6e/log/primary/logs/kolla/designate/designate-producer.txt | 14:16 |
| eandersson | In the new version of Designate the producer is critical path | 14:16 |
| eandersson | enabled_tasks = None | 14:17 |
| eandersson | https://zuul.opendev.org/t/openstack/build/b3a983600d6347a3abaa819461986d6e/log/primary/logs/kolla_configs/designate-producer/designate.conf#43 | 14:17 |
| eandersson | For some reason kolla is setting extensions to zero | 14:17 |
| eandersson | So the zone serial number is never incremented | 14:17 |
| eandersson | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/designate/templates/designate.conf.j2#L52 | 14:26 |
| eandersson | I'll throw in a quick PR to fix it | 14:26 |
| eandersson | frickler do you have a bug, or could you create one? https://review.opendev.org/c/openstack/kolla-ansible/+/892427 | 14:29 |
| frickler | eandersson: I wanted to, but got distracted by other issues, will add one later. thanks for spotting this | 14:32 |
| frickler | seems like that line has been in kolla forever, though, any idea what makes this break only now? | 14:33 |
| eandersson | I updated the commit message | 14:33 |
| eandersson | We moved the zone serial from central to the producer | 14:33 |
| eandersson | incrementing the zone serial | 14:34 |
| eandersson | So producer is critical path now, and wasn't before. | 14:35 |
| frickler | ah, that makes sense, thx | 14:36 |
| opendevreview | Michael Johnson proposed openstack/designate master: Fix DB upgrade with using SQLAlchemy 2.x https://review.opendev.org/c/openstack/designate/+/892543 | 14:36 |
| eandersson | Worth mentioning that the kolla issue is a bug too because while the producer wasn't critical path ebfore | 14:39 |
| eandersson | it had a lot of important recovery and cleanup functionality that ideally shouldn't be deactivated | 14:40 |
| frickler | eandersson: ah, so the issue was triggered when we switched kolla from deploying 2023.1 to master, I missed that patch during my search for possible triggers https://review.opendev.org/c/openstack/kolla/+/886087 | 14:40 |
| frickler | eandersson: yes, I agree, will create the bug now | 14:40 |
| eandersson | Yep - that makes sense | 14:43 |
| frickler | actually I found https://bugs.launchpad.net/kolla-ansible/+bug/1773972 with already describes a similar failure | 14:43 |
| frickler | eh, wrong link, sorry, https://bugs.launchpad.net/kolla-ansible/+bug/1879557 | 14:43 |
| frickler | eandersson: so is it enough to simply leave enabled_tasks undefined and thus using the default, or would it be better to make it configurable? | 14:45 |
| eandersson | I prefer undefined, because there are almost no scenarios where you would want to disable any of these jobs | 14:45 |
| eandersson | In the future we might add more optional jobs and it might make sense to make it configurable | 14:45 |
| eandersson | Having that said if you have some free cycles it probably does not hurt to have it configurable, but defaults to all tasks enabled | 14:47 |
| eandersson | (with a good comment) | 14:48 |
| eandersson | "Only disable tasks if you know what you are doing :D" | 14:48 |
| frickler | but I'm also confused after reading the docs, they say "enabled_tasks = None" is the default. so why does that config line change the default behavior | 14:48 |
| frickler | or is that string "None" vs. python None? | 14:48 |
| eandersson | Good question | 14:49 |
| eandersson | https://github.com/openstack/designate/blob/master/designate/conf/producer.py#L58C22-L58C22 | 14:49 |
| eandersson | You are right it should work, but might be something like Python None | 14:50 |
| eandersson | I can run it in a debugger real qucik | 14:50 |
| frickler | it is a list opt, so default = [] I guess | 14:50 |
| frickler | and setting = None in the config should give ["None"] then | 14:51 |
| eandersson | And no unit test to cover this code :D | 14:56 |
| eandersson | I'll add one real quick | 14:58 |
| eandersson | Yea - you are right | 15:09 |
| eandersson | <class 'list'> ['None'] | 15:09 |
| eandersson | So the "way" to set an empty list is just | 15:10 |
| eandersson | > enabled_tasks = | 15:10 |
| eandersson | Very confusing :D | 15:16 |
| -opendevstatus- NOTICE: Gerrit is going to be restarted to pick up a small config update. You will notice a short outage of the service. | 15:32 | |
| opendevreview | Merged openstack/designate master: Fix DB upgrade with using SQLAlchemy 2.x https://review.opendev.org/c/openstack/designate/+/892543 | 16:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!