| eandersson | The thing is that it's all super standard stuff and the calls are super inexpensive. It's in fact what I suspected. That the calls are so inexpensive and fast that it was pushing them out too fast (and maybe uwsgi had some internal rate limit aka ddos protection, or some sort of resource limitation). | 01:04 |
|---|---|---|
| *** hamalq has quit IRC | 01:32 | |
| eandersson | johnsom it just failed with the error that made me revert | 04:46 |
| johnsom | Ha, ok, I will dig in tomorrow | 04:47 |
| eandersson | https://d4f7229cf537dfa7a718-5d536abf72aff82bfe9baaa706f73d7c.ssl.cf2.rackcdn.com/785389/1/check/designate-bind9/c7766fc/testr_results.html | 04:47 |
| eandersson | Sounds good - thanks! | 04:47 |
| *** eandersson has quit IRC | 06:15 | |
| *** eandersson has joined #openstack-dns | 06:16 | |
| *** ChanServ has quit IRC | 07:58 | |
| *** timsim has quit IRC | 07:59 | |
| *** mugsie has quit IRC | 07:59 | |
| *** trident has quit IRC | 07:59 | |
| *** ircuser-1 has quit IRC | 07:59 | |
| *** johanssone has quit IRC | 07:59 | |
| *** yankcrime has quit IRC | 07:59 | |
| *** gmann has quit IRC | 07:59 | |
| *** nicolasbock has quit IRC | 07:59 | |
| *** jrosser has quit IRC | 07:59 | |
| *** PrinzElvis has quit IRC | 07:59 | |
| *** bbezak has quit IRC | 07:59 | |
| *** eandersson has quit IRC | 07:59 | |
| *** logan- has quit IRC | 07:59 | |
| *** zigo has quit IRC | 07:59 | |
| *** mgagne has quit IRC | 07:59 | |
| *** rektide has quit IRC | 07:59 | |
| *** johnsom has quit IRC | 07:59 | |
| *** k-s-dean has quit IRC | 07:59 | |
| *** openstackgerrit has quit IRC | 07:59 | |
| *** oklhost has quit IRC | 07:59 | |
| *** frickler has quit IRC | 07:59 | |
| *** ianychoi has quit IRC | 07:59 | |
| *** icey has quit IRC | 07:59 | |
| *** tobberydberg has quit IRC | 07:59 | |
| *** jmccrory has quit IRC | 07:59 | |
| *** cosmicsound has quit IRC | 07:59 | |
| *** andreaf has quit IRC | 07:59 | |
| *** Anticimex has quit IRC | 07:59 | |
| *** beagles has quit IRC | 07:59 | |
| *** bnemec has quit IRC | 07:59 | |
| *** swegener has quit IRC | 07:59 | |
| *** rm_work has quit IRC | 07:59 | |
| *** lbragstad has quit IRC | 07:59 | |
| *** michchap has quit IRC | 07:59 | |
| *** yankcrime has joined #openstack-dns | 08:04 | |
| *** johanssone has joined #openstack-dns | 08:04 | |
| *** ircuser-1 has joined #openstack-dns | 08:04 | |
| *** trident has joined #openstack-dns | 08:04 | |
| *** ChanServ has joined #openstack-dns | 08:04 | |
| *** verne.freenode.net sets mode: +o ChanServ | 08:04 | |
| *** timsim has joined #openstack-dns | 08:04 | |
| *** mugsie has joined #openstack-dns | 08:04 | |
| *** lbragstad has joined #openstack-dns | 08:05 | |
| *** michchap has joined #openstack-dns | 08:05 | |
| *** cosmicsound has joined #openstack-dns | 08:05 | |
| *** andreaf has joined #openstack-dns | 08:05 | |
| *** Anticimex has joined #openstack-dns | 08:05 | |
| *** rm_work has joined #openstack-dns | 08:05 | |
| *** jrosser has joined #openstack-dns | 08:06 | |
| *** nicolasbock has joined #openstack-dns | 08:06 | |
| *** gmann has joined #openstack-dns | 08:06 | |
| *** bbezak has joined #openstack-dns | 08:06 | |
| *** PrinzElvis has joined #openstack-dns | 08:06 | |
| *** jmccrory has joined #openstack-dns | 08:06 | |
| *** tobberydberg has joined #openstack-dns | 08:06 | |
| *** frickler has joined #openstack-dns | 08:06 | |
| *** ianychoi has joined #openstack-dns | 08:06 | |
| *** icey has joined #openstack-dns | 08:06 | |
| *** eandersson has joined #openstack-dns | 08:06 | |
| *** logan- has joined #openstack-dns | 08:06 | |
| *** zigo has joined #openstack-dns | 08:06 | |
| *** mgagne has joined #openstack-dns | 08:06 | |
| *** rektide has joined #openstack-dns | 08:06 | |
| *** johnsom has joined #openstack-dns | 08:06 | |
| *** k-s-dean has joined #openstack-dns | 08:06 | |
| *** oklhost has joined #openstack-dns | 08:06 | |
| *** openstackgerrit has joined #openstack-dns | 08:06 | |
| *** verne.freenode.net sets mode: +v openstackgerrit | 08:06 | |
| *** jrosser has quit IRC | 08:07 | |
| *** jrosser has joined #openstack-dns | 08:10 | |
| *** jrosser has quit IRC | 08:20 | |
| *** jrosser has joined #openstack-dns | 08:20 | |
| *** beagles has joined #openstack-dns | 08:49 | |
| *** swegener has joined #openstack-dns | 08:49 | |
| *** bnemec has joined #openstack-dns | 08:49 | |
| *** k-s-dean has quit IRC | 16:23 | |
| openstackgerrit | Michael Johnson proposed openstack/designate master: Revert "Move back to mod_wsgi for devstack" https://review.opendev.org/c/openstack/designate/+/785389 | 18:44 |
| johnsom | Ok, so my analysis of this is a worker starvation. The devstack default for uwsgi is two processes each with one thread. | 18:45 |
| johnsom | I bumped that to four processes each with one thread in the zuul job definition. | 18:45 |
| johnsom | Unfortunately devstack doesn't give us a lot of knobs to turn here. (no thread setting for uwsgi for example) | 18:46 |
| johnsom | Also a bummer, designate is using the same setting for all of it's process scaling as well. | 18:47 |
| johnsom | https://codesearch.opendev.org/?q=API_WORKERS&i=nope&files=&excludeFiles=&repos=openstack/designate | 18:47 |
| johnsom | So, we may need to do some other tuning to not eat a bunch of memory | 18:48 |
| johnsom | I posted an update to give it a go | 18:48 |
| eandersson | Probably worth fixing. I think I just took the lazy path there because orignally zuul only ever used a single worker for all Designate services. | 19:06 |
| openstackgerrit | Arkady Shtempler proposed openstack/designate-tempest-plugin master: Improving "_create_client_recordset" function https://review.opendev.org/c/openstack/designate-tempest-plugin/+/785720 | 19:19 |
| johnsom | Well, that didn't work, so that wasn't the issue. | 21:47 |
| johnsom | The key issue is https://zuul.opendev.org/t/openstack/build/4904d62e159c48088c913b2b14a97fc9/log/controller/logs/apache/error_log.txt#57 | 21:55 |
| johnsom | Something is breaking between apache and uwsgi. mod_proxy_uwsgi is throwing the error: | 22:05 |
| johnsom | https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_uwsgi.c?view=markup#l110 | 22:05 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!