| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 03:50 |
|---|---|---|
| johnsom | Let's step this debug up a bit.... | 03:51 |
| eandersson | I did find at least one very similar bug in uwsgi | 04:05 |
| eandersson | e.g. https://github.com/unbit/uwsgi/issues/835 | 04:06 |
| eandersson | but I am still unable to reproduce it at all locally. | 04:06 |
| johnsom | Yeah, 2015? | 04:06 |
| johnsom | Yeah, we perfectly reproduced that in our gate. | 04:07 |
| eandersson | It's weird that I cannot reproduce it locally. | 04:07 |
| johnsom | I guess I messed something up in the ansible, the jobs are bombing. looking at that now | 04:07 |
| eandersson | I don't understand what the difference can be. | 04:07 |
| johnsom | Something strange is happening with the socket. That last log shows the API returning 403 for a third time, but apache switches it to 502. The apache log even shows it got the 403, just as a secondary response. | 04:08 |
| johnsom | It seems like it's a timing thing, but I'm looking for more proof by dumping the socket traffic | 04:09 |
| eandersson | I wonder if it is hitting like a EAGAIN or EWOULDBLOCK and not handling it properly. | 04:10 |
| johnsom | EAGAIN appears to be handled in the mod_uwsgi, I don't see wouldblock though | 04:10 |
| eandersson | I think in theory they are the same | 04:11 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 04:12 |
| johnsom | It has been a while since I did raw socket work, so... rusty | 04:12 |
| johnsom | This is where mod_uwsgi is going off the rails: https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_uwsgi.c?view=markup#l407 | 04:13 |
| eandersson | I don't think I have ever had to go beyond just using standard Linux tooling for network :D | 04:14 |
| johnsom | Ah, well... Been there, done that | 04:14 |
| eandersson | I would love to know what rv is returning here | 04:20 |
| eandersson | It does not look like it is logged anywhere | 04:20 |
| johnsom | I captured it before I got distracted. I think my build of apache logs it | 04:21 |
| eandersson | https://apr.apache.org/docs/apr/1.6/group___a_p_r___s_t_a_t_u_s___i_s.html#ga56aa0a70756b5e83bc9d90f920527be4 | 04:21 |
| johnsom | geez, that was May. sigh. | 04:25 |
| johnsom | I think I will let this run, if it doesn't have what we want, I can spin a new apache and get it. | 04:26 |
| eandersson | > retry=0ProxyPass | 04:37 |
| eandersson | https://zuul.opendev.org/t/openstack/build/97438023fbb54e0c80093f82d6f56fbb/log/controller/logs/apache_config/designate-api-wsgi_conf.txt | 04:38 |
| johnsom | My ansible foo fails me | 04:38 |
| eandersson | Ah I guess it appended and didn't overwrite the config file | 04:38 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 04:42 |
| johnsom | Catch you all tomorrow | 05:12 |
| eandersson | o/ | 05:13 |
| eandersson | johnsom the issue is that it writes the file before devstack. It is actually devstack that is appending to the file | 05:28 |
| johnsom | Bummer, well feel free to poke at it, I won’t until tomorrow | 05:36 |
| opendevreview | Dr. Jens Harbott proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 05:49 |
| frickler | eandersson: johnsom: ^^ something like that I hope, overriding the config in the devstack plugin instead of the pre-playbook | 05:51 |
| eandersson | Yep that worked | 06:57 |
| eandersson | But now the endpoint isnt working properly | 06:57 |
| eandersson | > uwsgi: attempt to connect to Unix domain socket /tmp/designate-api-wsgi.socket (uwsgi-uds-designate-api-wsgi) failed | 06:58 |
| eandersson | guessing the socat piece isnt working properly | 07:03 |
| eandersson | but not really seeing anything in the logs | 07:03 |
| frickler | eandersson: I'm guessing that it is the /tmp isolation for apache mentioned in https://opendev.org/openstack/devstack/src/branch/master/lib/apache#L261 | 07:17 |
| frickler | let me try to move that to /var/run | 07:17 |
| opendevreview | Dr. Jens Harbott proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 07:36 |
| opendevreview | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Adds scenario based test case https://review.opendev.org/c/openstack/designate-tempest-plugin/+/816356 | 13:14 |
| opendevreview | Vadym Markov proposed openstack/designate-dashboard master: Fix policy request for Zones panel https://review.opendev.org/c/openstack/designate-dashboard/+/816189 | 15:28 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 16:40 |
| johnsom | I rebased apache, added a log for the rv and dumped socat's stderr/stdout to a file (though It should go to a log file I think). | 16:45 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 17:07 |
| johnsom | Sigh, learning the same lessons as months ago. | 18:17 |
| eandersson | Appreciate the effort. Logging etc will be a hundred times better if we get this working. | 18:23 |
| eandersson | You could probably write a blog post on this lol | 18:23 |
| johnsom | Ha, yah | 18:24 |
| johnsom | My fear is that it's just something dumb as we don't see this in other projects. Octavia has been using uwsgi for years... | 18:24 |
| eandersson | Yea - and there isn't anything special with this implementation and it is failing on the simplest of all calls. | 18:26 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 18:33 |
| johnsom | But, given there is an open bug since 2015, it might really be something.... | 18:34 |
| johnsom | I forgot that the trunk branch of apache has new requirements to build, I had to pull it back to the version installed in the base image. It should compile now | 18:35 |
| eandersson | Yea - might be worth to bump that thread | 18:52 |
| eandersson | Since we can easily reproduce this | 18:52 |
| eandersson | I wonder if we are the only ones doing a million of these calls to test all APIs for 403s | 18:56 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 21:54 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 22:33 |
| opendevreview | Michael Johnson proposed openstack/designate master: DNM: Debug uWSGI https://review.opendev.org/c/openstack/designate/+/786506 | 23:17 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!