| eandersson | Yea - gonna give it a few more retries, but don't think we can reproduce it at the moment | 00:53 |
|---|---|---|
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Revert "Move back to mod_wsgi for devstack" https://review.opendev.org/c/openstack/designate/+/785389 | 01:47 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 01:47 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Fix dns.query.tcp/udp not always handling ipv6 properly https://review.opendev.org/c/openstack/designate/+/813722 | 02:25 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/designate master: Fix dns.query.tcp/udp not always handling ipv6 properly https://review.opendev.org/c/openstack/designate/+/813722 | 02:32 |
| eandersson | At least this proves that the issue isn’t with our code, but a bug with uwsgi | 05:20 |
| noonedeadpunk | hey there! Have a question about designate-sink. How it's designed to get notifications from nova/neutron queues? I don't see in code or config anithyng that would allow to define rabbit credentials | 12:45 |
| noonedeadpunk | Like in scenario when neutron/nova use their own vhosts or even separate rabbitmq clusters from designate | 12:45 |
| noonedeadpunk | as it feels like designate expects notifications to be present inside it's own vhost/cluster? | 12:46 |
| noonedeadpunk | oh, just found https://specs.openstack.org/openstack/oslo-specs/specs/victoria/support-transports-per-oslo-notifications.html which I guess answers my question | 12:50 |
| johnsom | eandersson So no hits overnight? Any comments before I start removing some of the debug parts and reproduce that way? | 15:53 |
| eandersson | Nah I think we are good to go. I tried reducing the uwsgi workers to 1 just to see if that would help. | 16:55 |
| johnsom | I think the next step is backing out of the debug setup. Starting with pulling out the socat tap | 18:03 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 19:06 |
| eandersson | I changed WORKER count from 4 -> 1 as well. Might want to bump that back up if it does not fail. | 19:13 |
| johnsom | Yeah, I was thinking the same | 19:30 |
| johnsom | eanderson First run after taking socat out.... | 20:06 |
| eandersson | wow yea | 20:22 |
| eandersson | I assume its this right 19:44:18.000374 ? | 20:42 |
| johnsom | I was looking at 19:44:21.176392 initially | 20:57 |
| johnsom | I think there are two calls looking at the access log | 20:58 |
| eandersson | Is 70014 = APR_STATUS_IS_EINTR ? | 21:37 |
| johnsom | APR_EOF I think | 21:37 |
| eandersson | Ah yea that adds up to EOF | 21:39 |
| eandersson | Not sure why the documentation doesnt just add it up for you and display the int value :p | 21:40 |
| johnsom | Yeah, or useful comments in the apr code | 21:45 |
| johnsom | It's Tue Nov 09 19:44:18.679918 2021 | 23:03 |
| eandersson | johnsom what is 104? | 23:05 |
| johnsom | ECONNRESET I think | 23:08 |
| johnsom | Basically reading from a closed socket | 23:09 |
| johnsom | This is what I saw back in may | 23:10 |
| johnsom | Well, my "not an apache dev" guess would be https://github.com/johnsom/httpd/blob/trunk/modules/proxy/mod_proxy_uwsgi.c#L401 should be checking for APR_ECONNRESET as well. | 23:27 |
| eandersson | Yea - I feel like something is missing. | 23:39 |
| eandersson | Easy enough for you to add I guess and retry | 23:39 |
| johnsom | Yeah, I am dumpster diving for bug reports just to see if I get lucky | 23:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!