| *** dougbtv_ has joined #ara | 01:13 | |
| *** dougbtv has quit IRC | 01:15 | |
| *** harlowja has quit IRC | 01:19 | |
| ara-slack | <dmsimard> @harlowja hey how many threads/processes/workers are you running gunicorn with ? | 03:13 |
|---|---|---|
| *** harlowja has joined #ara | 05:16 | |
| ara-slack | <harlowja> 2 workers with 2 threads per | 05:19 |
| *** harlowja has quit IRC | 05:20 | |
| ara-slack | <harlowja> File uploaded https://ara-community.slack.com/files/U6V9WRRTM/FA04GGX7A/-.txt / https://slack-files.com/T6VAB05L7-FA04GGX7A-141acae540 - the max-requests really seems to help | 05:20 |
| ara-slack | <harlowja> but its not really a fix :(] | 05:46 |
| *** Brainspackle has quit IRC | 05:54 | |
| *** Brainspackle has joined #ara | 05:57 | |
| openstackgerrit | Ian Wienand proposed openstack/ara-clients master: [DNM] testing https://review.openstack.org/558754 | 08:51 |
| openstackgerrit | Ian Wienand proposed openstack/ara-clients master: [DNM] testing https://review.openstack.org/558754 | 08:56 |
| *** boxrick_ has quit IRC | 09:39 | |
| *** boxrick has joined #ara | 09:40 | |
| *** cliles has joined #ara | 10:10 | |
| *** weshay_ has joined #ara | 10:14 | |
| *** cliles_ has quit IRC | 10:15 | |
| *** jrist has quit IRC | 10:15 | |
| *** weshay has quit IRC | 10:15 | |
| *** dmsimard has quit IRC | 10:15 | |
| *** dmsimard has joined #ara | 10:17 | |
| *** jrist has joined #ara | 10:20 | |
| *** weshay_ has quit IRC | 10:28 | |
| *** weshay_ has joined #ara | 10:28 | |
| ara-slack | <dmsimard> @harlowja that's not enough. It made me think about that because I just submitted a patch for that upstream: https://review.openstack.org/#/c/558687/ | 11:45 |
| *** tbielawa has joined #ara | 11:46 | |
| ara-slack | <dmsimard> The default rule of thumb for mod_wsgi is one thread and processes = <number of cores> | 11:46 |
| ara-slack | <dmsimard> See the impact on the process CPU usage when I bump the amount of processes: https://m.imgur.com/a/e2YFR | 11:47 |
| ara-slack | <dmsimard> It's also much more snappy (and Apache returns HTTP 304's for static files.. which gunicorn probably doesn't do) | 11:49 |
| ara-slack | <dmsimard> Can you try playing with that kind of config and let me know if that helps performance/RAM usage ? | 11:50 |
| ara-slack | <dmsimard> Try something like 1 thread but 8 workers | 11:51 |
| dmsimard | There's a thread that talks about threads/processes but I can't find it right now.. this one is good though: http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html | 12:04 |
| *** weshay_ is now known as weshay | 13:06 | |
| *** bcoca has joined #ara | 13:45 | |
| *** bcoca has quit IRC | 13:45 | |
| *** bcoca has joined #ara | 13:45 | |
| *** tbielawa has quit IRC | 14:51 | |
| *** rvgate has joined #ara | 15:25 | |
| *** tbielawa has joined #ara | 15:28 | |
| *** evrardjp_ has joined #ara | 15:35 | |
| *** evrardjp has quit IRC | 15:36 | |
| ara-slack | <harlowja> @dmsimard yes i know its not enough | 15:57 |
| ara-slack | <harlowja> but when a worker can consume like 2 GB in a 8 GB VM :S | 15:57 |
| ara-slack | <harlowja> 2 * 2 * 2 == 8, lol | 15:57 |
| ara-slack | <harlowja> i can try 1 thread and 8 workers but that would hit 8 * 2GB (at some times) | 15:58 |
| ara-slack | <harlowja> and then crash, lol | 15:58 |
| *** evrardjp_ is now known as evrardjp | 16:05 | |
| ara-slack | <dmsimard> @harlowja that's not necessarily true | 16:16 |
| ara-slack | <dmsimard> anyway, try it.. if it goes oom and it's not faster then I guess it wasn't that | 16:16 |
| ara-slack | <dmsimard> just saying at your scale you need more threads/processes than that, especially more so if you don't have a web server to handle the static things | 16:17 |
| ara-slack | <harlowja> agreed | 16:17 |
| ara-slack | <dmsimard> IMO you really need a webserver in front, even if it's just an nginx reverse proxy to gunicorn | 16:18 |
| ara-slack | <dmsimard> and see if tweaking those threads/processes help | 16:18 |
| ara-slack | <dmsimard> I don't know what's the nomenclature/vocabulary in gunicorn but in apache's mod_wsgi, you typically want one thread but N processes | 16:19 |
| ara-slack | <harlowja> yup | 16:19 |
| ara-slack | <harlowja> i agree | 16:19 |
| *** tbielawa is now known as tbielawa|brb | 17:04 | |
| *** tbielawa|brb has quit IRC | 17:09 | |
| ara-slack | <dmsimard> @harlowja another thing to try: autocreate_database = false | 17:38 |
| dmsimard | there's a few patches coming that are performance oriented | 17:39 |
| dmsimard | I'll also find a way to backport two patches from 1.0 which should yield performance improvements | 17:39 |
| dmsimard | any fairly large scale user want to try something out ? I want to see if there's a noticeable perf improvement for something | 17:41 |
| ara-slack | <harlowja> ah | 17:59 |
| ara-slack | <harlowja> will try that | 17:59 |
| ara-slack | <harlowja> i can try 1.0 | 18:00 |
| ara-slack | <harlowja> i think | 18:00 |
| *** tbielawa has joined #ara | 18:03 | |
| *** spiette has quit IRC | 18:16 | |
| *** spiette has joined #ara | 18:19 | |
| ara-slack | <dmsimard> @harlowja don't try the 1.0 branch, I'll backport what I want from it | 18:26 |
| *** tbielawa is now known as tbielawa|brb | 18:28 | |
| *** tbielawa|brb has quit IRC | 18:32 | |
| *** tbielawa has joined #ara | 18:42 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:51 | |
| ara-slack | <harlowja> kk | 18:59 |
| ara-slack | <harlowja> we can deploy automatically; so not superhard to test out things | 18:59 |
| ara-slack | <harlowja> as i add in `autocreate_database =false` | 18:59 |
| ara-slack | <harlowja> though doing ^ will mean someone/something has to create the initial database, lol | 19:00 |
| ara-slack | <harlowja> File uploaded https://ara-community.slack.com/files/U6V9WRRTM/FA12T5Q68/-.txt / https://slack-files.com/T6VAB05L7-FA12T5Q68-c5777c1e8c - gonna bump it up to 8 (*2threads) workers to, see how it goes | 19:05 |
| ara-slack | <harlowja> `--max-requests 25` i think really helped | 19:08 |
| ara-slack | <harlowja> but imho that just means something is holding onto way to much memory | 19:08 |
| ara-slack | <harlowja> or that somehow during one of these sqlalachemy usages it loads way more than it needs | 19:08 |
| ara-slack | <harlowja> like maybe https://github.com/openstack/ara/blob/master/ara/views/reports.py#L59 is just grabbing everything | 19:09 |
| *** harlowja has joined #ara | 19:10 | |
| ara-slack | <harlowja> or something else like that which would cause the non-resident ram to explode | 19:10 |
| ara-slack | <dmsimard> Your database already exists, no ? It doesn't need to be created or verified if migrations need to be applied. | 19:32 |
| ara-slack | <dmsimard> I'll try and see if I'm able to reproduce your issue using the fake 1000 hosts I've got | 19:34 |
| *** myoung is now known as myoung|mtg | 20:10 | |
| dmsimard | backporting 1.0 stuff into master is... fun | 20:22 |
| *** myoung|mtg is now known as myoung | 21:03 | |
| *** tbielawa has quit IRC | 21:45 | |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Disable database automatic creation/migration in the sqlite middleware https://review.openstack.org/558946 | 22:11 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add a bit of verbose debug to the sqlite middleware https://review.openstack.org/558947 | 22:11 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: 1.0 backport: Add helper modules/scripts to know where ARA is https://review.openstack.org/558948 | 22:12 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: 1.0 backport: Refactor configuration and instanciate it https://review.openstack.org/558949 | 22:12 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Fix broken unit tests with the release of Ansible 2.5 https://review.openstack.org/558957 | 22:47 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: 1.0 backport: Add helper modules/scripts to know where ARA is https://review.openstack.org/558948 | 22:54 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: 1.0 backport: Refactor configuration and instanciate it https://review.openstack.org/558949 | 22:54 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Disable database automatic creation/migration in the sqlite middleware https://review.openstack.org/558946 | 22:54 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add a bit of verbose debug to the sqlite middleware https://review.openstack.org/558947 | 22:54 |
| dmsimard | bcoca: the only thing that 2.5 broke was unit tests, good job :) | 22:55 |
| dmsimard | i.e, https://review.openstack.org/#/c/558957/1/ara/tests/unit/test_ara_record.py | 22:55 |
| ara-slack | <harlowja> File uploaded https://ara-community.slack.com/files/U6V9WRRTM/FA2F1BXF1/screen_shot_2018-04-04_at_3.57.33_pm.png / https://slack-files.com/T6VAB05L7-FA2F1BXF1-0869eafaca - the max-requests stuff really seems to be helping | 22:58 |
| ara-slack | <dmsimard> @harlowja so the threads were getting hung up or something ? | 23:47 |
| ara-slack | <harlowja> or just holding onto memory | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
