| frickler | eandersson: johnsom: I wonder how can there be a messaging timeout in designate unit testing? do we really use messaging there or is that just a missing mock? https://zuul.opendev.org/t/openstack/build/955be355999b45a1bed5213ad7b7620a | 12:38 |
|---|---|---|
| opendevreview | Merged openstack/designate master: Add mysql_engine and mysql_charset options to shared_zones table https://review.opendev.org/c/openstack/designate/+/893344 | 13:59 |
| eandersson | frickler: It's the fake rmq driver | 14:02 |
| eandersson | > "/home/zuul/src/opendev.org/openstack/designate/.tox/py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py" | 14:02 |
| eandersson | The functional unit tests talks between each other using the fake oslo.messaging driver e.g. API service talking to Central | 14:03 |
| frickler | eandersson: hmm, but how can that generate a timeout? or isn't that the real issue for the failure? | 14:07 |
| eandersson | The fake driver probably adhers to the same timeouts as the real driver, so central was most likely either super slow, or stuck in a deadlock | 14:08 |
| eandersson | So I would say legit failure, but difficult to tell what really happened as central generates no logs :'( | 14:08 |
| eandersson | https://github.com/openstack/designate/blob/360433b38b449d1c53ab1357fdb0c4608c09efa5/designate/tests/__init__.py#L342 | 14:09 |
| eandersson | In theory just a really slow or overloaded system could cause this, assuming it took longer than 5 seconds to do something | 14:09 |
| johnsom | At the end of the cycle, some nodepool hosts get really slow on IO, we see it every time in Octavia job run times. We have done some investigation with infra, and it seems like the swapping many jobs have can overload the IO on some of the nodepool hosts. | 14:59 |
| frickler | maybe bump the 5 seconds a bit if we see this repeatedly? | 15:45 |
| frickler | anyway thx for explaining eandersson | 15:45 |
| eandersson | I haven't seen it in years, so hopefully not something that becomes more frequent | 15:58 |
| eandersson | (but totally open to bumping it if it helps with stability) | 16:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!