| ara-slack | <dmsimard> @logan.attwood can you give a try installing from master and let me know if the memory looks better ? | 00:03 |
|---|---|---|
| ara-slack | <dmsimard> i.e, pip install git+https://github.com/openstack/ara | 00:04 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add support for configuring sqlalchemy pool size, timeout and recycle https://review.openstack.org/559609 | 02:09 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add support for configuring sqlalchemy pool size, timeout and recycle https://review.openstack.org/524427 | 02:10 |
| dmsimard | A late shower thought: there's probably nothing stopping us from doing like the sqlite middleware but with mysql/postgresql instead | 02:55 |
| dmsimard | For example... a role that creates a database on a mysql server just for that run. | 02:56 |
| dmsimard | So each playbook run would end up being it's own database ? That would make it easy to shard I guess ? | 02:56 |
| dmsimard | https://github.com/openstack-infra/puppet-openstackci/blob/7a4e625ca157fc9480574f439d7463356f4692e3/templates/logs-dev.vhost.erb#L66-L91 | 02:58 |
| dmsimard | lookups would stay fast because there's no enormous amount of rows to scan | 02:58 |
| dmsimard | There might be a lot of databases though ? I've never ran a server with a LOT of databases -- fewer but larger. Does that work ? | 02:59 |
| mnaser | i feel like that is "yet another component i would have to maintain" | 03:00 |
| dmsimard | ara could take care of it | 03:01 |
| dmsimard | but I mean it wouldn't be required | 03:01 |
| dmsimard | shard_databases = true | 03:01 |
| dmsimard | or shard_reports = true | 03:01 |
| dmsimard | it's a scalability and performance thing | 03:02 |
| mnaser | i think it depends on the use case | 03:02 |
| mnaser | i think i'm viewing it from the zuul/ci side of things | 03:02 |
| dmsimard | zuul already has a mysql server | 03:02 |
| dmsimard | eh | 03:02 |
| mnaser | probably not enough to handle 1000+ databases a day | 03:03 |
| mnaser | :p | 03:03 |
| dmsimard | well | 03:03 |
| dmsimard | it's not like they are big databases ? | 03:03 |
| mnaser | i'd feel it's still a significant amount of hit on that server, i think. especially i guess with all the output that some jobs can generate | 03:03 |
| dmsimard | I dunno. Like I wonder what would happen if we created like 20 000 databases every day | 03:03 |
| dmsimard | maybe mysql is not the right thing | 03:03 |
| dmsimard | I wanna say mongodb but meh | 03:03 |
| mnaser | i think like | 03:04 |
| mnaser | rendering static json files simulating the api maybe :p | 03:04 |
| dmsimard | lol | 03:04 |
| dmsimard | it's a nice hack | 03:04 |
| dmsimard | there are so many hacks in the frontend because I can't do frontend | 03:05 |
| mnaser | i think that would be a cool idea and would take off a huge load and avoid using sqlite, i mean the api is pretty static | 03:05 |
| dmsimard | the file system tree ? dude that was hard | 03:05 |
| mnaser | whatever that api response generates is always the same | 03:05 |
| dmsimard | ? | 03:05 |
| dmsimard | I don't follow | 03:05 |
| dmsimard | oh btw, when I meant multiple databases -- it's just not really different than how you'd have different sqlite databases | 03:21 |
| dmsimard | oh, the sqlite middleware should put something in the header to show where it's from. | 04:46 |
| dmsimard | especially since the database can be downloaded. | 04:46 |
| *** gvincent has joined #ara | 05:26 | |
| *** jparrill has joined #ara | 05:40 | |
| *** rvgate has joined #ara | 09:12 | |
| *** jrm_ has joined #ara | 09:26 | |
| *** sshnaidm|off is now known as sshnaidm | 09:30 | |
| jrm_ | hey, was there a resolution to: UndefinedError: 'macros' is undefined ? | 09:56 |
| *** rvgate has quit IRC | 10:31 | |
| dmsimard | jrm_: I loo | 11:37 |
| dmsimard | jrm_: looked but didn't remember where I saw that | 11:37 |
| dmsimard | What's your version of Ansible ? | 11:38 |
| dmsimard | And jinja ? | 11:38 |
| dmsimard | jrm_: oh wait, you were trying to get it working with AWX right ? I got that working the other day: https://mobile.twitter.com/ARA_Community/status/982355410890960897 | 11:41 |
| dmsimard | There seems to be a bug that makes this harder to set up than it should be though: https://github.com/ansible/awx/issues/1737 | 11:42 |
| *** rvgate has joined #ara | 12:06 | |
| *** hwoarang has joined #ara | 12:12 | |
| jrm_ | yup, that was me. So it's not so much an ARA thing as it is something terrible that happens in AWX? | 12:21 |
| jrm_ | it was AWX 1.0.5 and Jinja was 2.8 | 12:22 |
| dmsimard | jrm_: re: your macros issue: https://github.com/openstack/ara/commit/49d1702f3a5b6522bf0f97a3fd953b671b4bec81 | 12:54 |
| *** hwoarang has quit IRC | 12:57 | |
| *** hwoarang has joined #ara | 12:57 | |
| *** tbielawa has joined #ara | 12:59 | |
| *** tbielawa has quit IRC | 13:07 | |
| *** tbielawa has joined #ara | 13:10 | |
| *** bcoca has joined #ara | 13:15 | |
| *** bcoca has joined #ara | 13:15 | |
| *** hwoarang has quit IRC | 13:35 | |
| *** tbielawa is now known as tbielawa|brb | 14:04 | |
| jrm_ | dmsimard: that seems to fix. Ubuntu 16.04LTS only has python-jinja2.8 in the official repos though. | 14:10 |
| dmsimard | yeah 2.9 dates back to january 2017, it's been out for a while but I guess distros might not have caught up yet | 14:12 |
| ara-slack | <logan.attwood> ack | 14:12 |
| ara-slack | <dmsimard> Thanks! Let me know how that goes | 14:14 |
| *** jrm_ has quit IRC | 14:16 | |
| *** tbielawa|brb is now known as tbielawa | 14:16 | |
| ara-slack | <logan.attwood> lots of errors :( | 14:27 |
| ara-slack | <dmsimard> uh oh | 14:27 |
| ara-slack | <logan.attwood> ``` ara_1 | Traceback (most recent call last): ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__ ara_1 | return self.wsgi_app(environ, start_response) ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1985, in wsgi_app ara_1 | response = self.handle_exception(e) ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1540, in | 14:27 |
| ara-slack | handle_exception ara_1 | reraise(exc_type, exc_value, tb) ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app ara_1 | response = self.full_dispatch_request() ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request ara_1 | rv = self.handle_user_exception(e) ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in | 14:27 |
| ara-slack | handle_user_exception ara_1 | reraise(exc_type, exc_value, tb) ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request ara_1 | rv = self.dispatch_request() ara_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request ara_1 | return | 14:27 |
| ara-slack | self.view_functions[rule.endpoint](**req.view_args) ara_1 | File "/usr/local/lib/python2.7/dist-packages/ara/views/reports.py", line 121, in ajax_plays ara_1 | if not utils.fast_count(plays): ara_1 | File "/usr/local/lib/python2.7/dist-packages/ara/utils.py", line 106, in fast_count ara_1 | count = query.session.execute(count_query).scalar() ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1176, i | 14:27 |
| ara-slack | execute ara_1 | bind, close_with_result=True).execute(clause, params or {}) ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 948, in execute ara_1 | return meth(self, multiparams, params) ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection ara_1 | return connection._execute_clauseelement(self, multiparams, params) ara_1 | File | 14:27 |
| ara-slack | "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement ara_1 | compiled_sql, distilled_params ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context ara_1 | context) ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception ara_1 | exc_info ara_1 | File | 14:27 |
| ara-slack | "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause ara_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause) ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context ara_1 | context) ara_1 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute ara_1 | cursor.execute(statement | 14:27 |
| ara-slack | parameters) ara_1 | DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac ``` | 14:27 |
| ara-slack | <dmsimard> that's odd | 14:27 |
| ara-slack | <logan.attwood> yeah | 14:27 |
| ara-slack | <dmsimard> were you still using that sqlalchemy pool patch ? | 14:27 |
| ara-slack | <logan.attwood> i had previously been installing from pypi | 14:27 |
| ara-slack | <logan.attwood> oh | 14:28 |
| ara-slack | <logan.attwood> never was | 14:28 |
| ara-slack | <dmsimard> that's fine | 14:28 |
| ara-slack | <logan.attwood> lemme make sure it doesn't happen with the latest version on pypi | 14:28 |
| ara-slack | <dmsimard> ok | 14:28 |
| ara-slack | <dmsimard> FWIW I'm working on rebasing the sqlalchemy pool patch https://review.openstack.org/#/c/524427/ | 14:28 |
| *** tbielawa is now known as tbielawa|mtg | 14:30 | |
| ara-slack | <logan.attwood> works fine... | 14:31 |
| ara-slack | <logan.attwood> w/ the pip one | 14:31 |
| ara-slack | <logan.attwood> well | 14:31 |
| ara-slack | <logan.attwood> other than the issue i previously had, xD | 14:31 |
| ara-slack | <logan.attwood> works fine under apache | 14:42 |
| ara-slack | <logan.attwood> so | 14:42 |
| ara-slack | <logan.attwood> https://github.com/openstack/ara/commit/680128e5f7f9c16bb0d8f1d09c1ebc6beee50bb5 | 14:42 |
| ara-slack | <dmsimard> with the patch yo mean ? | 14:42 |
| ara-slack | <logan.attwood> no | 14:42 |
| ara-slack | <logan.attwood> i mean running on master | 14:43 |
| ara-slack | <logan.attwood> works fine under apache | 14:43 |
| ara-slack | <dmsimard> yeah but was that not what you tried before ? | 14:43 |
| ara-slack | <logan.attwood> i had it configured for `runserver` mode when troubleshooting, when changing over to the install from master I left that alone thinking they'd be identical :slightly_smiling_face: | 14:43 |
| ara-slack | <logan.attwood> baaah | 14:44 |
| ara-slack | <logan.attwood> one of the larger ones died | 14:44 |
| ara-slack | <logan.attwood> all of the big ones died :( | 14:45 |
| ara-slack | <dmsimard> died ? | 14:46 |
| ara-slack | <logan.attwood> yeah | 14:46 |
| ara-slack | <logan.attwood> OOM | 14:46 |
| ara-slack | <logan.attwood> well | 14:47 |
| ara-slack | <dmsimard> with master ? | 14:47 |
| ara-slack | <logan.attwood> yes | 14:47 |
| ara-slack | <logan.attwood> i'm assuming it's OOM, don't see OOM when it runs under apache, only under `runserver`, and `runserver` doesn't work. | 14:47 |
| ara-slack | <dmsimard> right | 14:47 |
| ara-slack | <dmsimard> The OOM fix was in the wsgi implementation | 14:47 |
| ara-slack | <dmsimard> So you would not see it when using runserver | 14:47 |
| ara-slack | <dmsimard> Also using runserver at your scale is asking for pain and suffering, it's an embedded development server :slightly_smiling_face: | 14:48 |
| ara-slack | <logan.attwood> just for testing, and only locally :slightly_smiling_face: | 14:48 |
| ara-slack | <logan.attwood> i'm not a complete masochist. | 14:48 |
| ara-slack | <dmsimard> so you're saying it's fixed with apache then ? tbh I only tested with gunicorn to reproduce @harlowja's config | 14:49 |
| ara-slack | <logan.attwood> it OOMs under apache | 14:49 |
| ara-slack | <logan.attwood> it straight up doesn't work at all under runserver on master | 14:49 |
| ara-slack | <dmsimard> can you paste your vhost config ? | 14:49 |
| ara-slack | <dmsimard> I'll try and reproduce | 14:50 |
| ara-slack | <logan.attwood> ``` <VirtualHost *:80> ServerName ara WSGIDaemonProcess ara user=www-data group=www-data processes=4 threads=1 WSGIScriptAlias / /var/www/ara/ara-wsgi SetEnv ANSIBLE_CONFIG /var/www/ara/ansible.cfg <Directory /var/www/ara> WSGIProcessGroup ara WSGIApplicationGroup %{GLOBAL} Require all granted </Directory> </VirtualHost> # vim: syntax=apache ts=4 sw=4 sts=4 sr noet ``` | 14:50 |
| ara-slack | <logan.attwood> base http config ``` Mutex file:${APACHE_LOCK_DIR} default PidFile ${APACHE_PID_FILE} Timeout 300 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5 User ${APACHE_RUN_USER} Group ${APACHE_RUN_GROUP} HostnameLookups Off ErrorLog /dev/stdout LogLevel info CustomLog /dev/stdout combined IncludeOptional mods-enabled/*.load IncludeOptional mods-enabled/*.conf Include ports.conf <Directory /> Options FollowSymLinks | 14:50 |
| ara-slack | AllowOverride None Require all denied </Directory> <Directory /usr/share> AllowOverride None Require all granted </Directory> <Directory /var/www/> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> AccessFileName .htaccess <FilesMatch "^\.ht"> Require all denied </FilesMatch> LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" | 14:50 |
| ara-slack | vhost_combined LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %O" common LogFormat "%{Referer}i -> %U" referer LogFormat "%{User-agent}i" agent IncludeOptional conf-enabled/*.conf IncludeOptional sites-enabled/*.conf # vim: syntax=apache ts=4 sw=4 sts=4 sr noet ``` | 14:50 |
| ara-slack | <dmsimard> yeah that's pretty much straight what's in the docs | 14:50 |
| ara-slack | <dmsimard> ok, I'll try that | 14:50 |
| ara-slack | <logan.attwood> some apache logs ``` ara_1 | 172.21.0.1 - - [09/Apr/2018:14:48:15 +0000] "GET /reports/ajax/parameters/6a18a68a-289a-460d-97f6-1b97be2d499d.txt?_=1523285105111 HTTP/1.1" 200 1223 "http://localhost/repo rts/list/2.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" ara_1 | 172.21.0.1 - - [09/Apr/2018:14:48:06 +0000] "GET | 14:51 |
| ara-slack | /reports/ajax/stats/7257a931-e22c-4b8e-ad08-bc6948171ef9.txt?_=1523285105108 HTTP/1.1" 200 448502 "http://localhost/reports /list/2.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" ara_1 | 172.21.0.1 - - [09/Apr/2018:14:48:15 +0000] "GET /reports/ajax/stats/6a18a68a-289a-460d-97f6-1b97be2d499d.txt?_=1523285105112 HTTP/1.1" 200 449011 "http://localhost/reports /list/2.html | 14:51 |
| ara-slack | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" ara_1 | [Mon Apr 09 14:49:47.402913 2018] [wsgi:error] [pid 33:tid 140421293545216] [client 172.21.0.1:55262] Truncated or oversized response headers received from daemon process ' ara': /var/www/ara/ara-wsgi, referer: http://localhost/reports/list/2.html ara_1 | 172.21.0.1 - - [09/Apr/2018:14:48:05 +0000] "GET | 14:51 |
| ara-slack | /reports/ajax/results/cabf5f4c-20d6-4477-865e-8996138d2588.txt?_=1523285105102 HTTP/1.1" 500 798 "http://localhost/reports/ list/2.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" ara_1 | [Mon Apr 09 14:49:47.865006 2018] [wsgi:info] [pid 27:tid 140421406017408] mod_wsgi (pid=116): Process 'ara' has died, deregister and restart it. ara_1 | [Mon Apr 09 14:49:47.866224 201 | 14:51 |
| ara-slack | [wsgi:info] [pid 27:tid 140421406017408] mod_wsgi (pid=116): Process 'ara' terminated by signal 9 ara_1 | [Mon Apr 09 14:49:47.866525 2018] [wsgi:info] [pid 27:tid 140421406017408] mod_wsgi (pid=116): Process 'ara' has been deregistered and will no longer be monitored. ara_1 | [Mon Apr 09 14:49:47.871889 2018] [wsgi:info] [pid 126:tid 140421406017408] mod_wsgi (pid=126): Starting process 'ara' with uid=33, gid=33 and threads=1. ara_1 | [Mon Apr 09 | 14:51 |
| ara-slack | 14:49:47.877445 2018] [wsgi:info] [pid 126:tid 140421406017408] mod_wsgi (pid=126): Initializing Python. ara_1 | [Mon Apr 09 14:49:47.921091 2018] [wsgi:info] [pid 126:tid 140421406017408] mod_wsgi (pid=126): Attach interpreter ''. ``` | 14:51 |
| ara-slack | <dmsimard> I stopped witnessing the memory leaks with gunicorn (it still takes up a bit of memory but doesn't keep growing), maybe there's something else | 14:51 |
| ara-slack | <dmsimard> how much ram is it using before running OOM ? | 14:51 |
| ara-slack | <logan.attwood> lemme get `doenter` running | 14:52 |
| *** spiette has joined #ara | 14:56 | |
| ara-slack | Action: logan.attwood watches it slowly climb | 14:57 |
| dmsimard | I have to step away for dentist appointment, be back in a bit | 14:59 |
| ara-slack | <logan.attwood> daaaamn. | 14:59 |
| ara-slack | <logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA31508LR/image.png / https://slack-files.com/T6VAB05L7-FA31508LR-30540615e4 | 14:59 |
| ara-slack | <logan.attwood> uhh | 14:59 |
| ara-slack | <logan.attwood> *all* the ram | 14:59 |
| ara-slack | <dmsimard> That's pretty | 14:59 |
| ara-slack | <logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA4JPLY0P/image.png / https://slack-files.com/T6VAB05L7-FA4JPLY0P-15cab61134 | 15:00 |
| ara-slack | <dmsimard> okay, so that means it's probably not fixed yet.. interesting. I'll look after the dentist. | 15:00 |
| ara-slack | <logan.attwood> ty | 15:00 |
| ara-slack | <logan.attwood> well | 15:00 |
| ara-slack | <logan.attwood> it's like | 15:00 |
| ara-slack | <logan.attwood> nothing for a efw minutes | 15:00 |
| ara-slack | <logan.attwood> then ramps up quickly | 15:00 |
| dmsimard | FWIW my attempt at fixing it was https://github.com/openstack/ara/commit/fa624de6f78ec637fd6f46524abffcddb632e784 | 15:00 |
| *** hwoarang has joined #ara | 15:14 | |
| *** tbielawa|mtg is now known as tbielawa | 15:32 | |
| *** Phinnay has joined #ara | 16:25 | |
| Phinnay | is there a way to access collected records from the CLI interface? | 16:26 |
| ara-slack | <harlowja> @dmsimard thanks for the info; let us know what u find out (i'd be intersting in knowing why it consumes memory like it does) | 16:27 |
| dmsimard | Phinnay: ara data list -b <playbook id> | 16:27 |
| dmsimard | Phinnay: (it should read record list but it's a long story) | 16:27 |
| Phinnay | aha ok | 16:28 |
| Phinnay | any thoughs on what was up with the problems saving records to MYsql from last week? | 16:36 |
| Phinnay | been solid into sqlite | 16:36 |
| dmsimard | Phinnay: it's still on my todo list, I've fixed other things but not that | 16:38 |
| Phinnay | alrighty | 16:39 |
| Phinnay | as long as it's on the list for the next release i'll hold tight just wanted to make sure it got entered as a bug | 16:39 |
| Phinnay | also am i missing something with ara playbook list -c and multiple column names? | 16:45 |
| ara-slack | <harlowja> @dmsimard let me know if u need any help also | 16:45 |
| ara-slack | <harlowja> or testing stuffs | 16:45 |
| dmsimard | Phinnay: you need to repeat -c for columns | 16:46 |
| Phinnay | (ara) playbook list -c 'ID','Path' No recognized column names in ['ID,Path']. Recognized columns are ['ID', 'Path', 'Time Start', 'Duration', 'Complete', 'Ansible Version']. | 16:46 |
| dmsimard | try -c ID -c Path | 16:46 |
| Phinnay | oh ok the use verbiage makes it seem like you can give it a list | 16:46 |
| Phinnay | ok that works fine :D | 16:47 |
| *** tbielawa is now known as tbielawa|lunch | 16:49 | |
| -openstackstatus- NOTICE: zuul was restarted to update to the latest code; please recheck any changes uploaded within the past 10 minutes | 16:50 | |
| *** Phinnay has quit IRC | 17:34 | |
| *** tbielawa|lunch is now known as tbielawa | 17:52 | |
| ara-slack | <dmsimard> @logan.attwood @harlowja I'm not able to go higher than ~1.5GB RAM with gunicorn against the current master with sqlite against 4 playboks involving >1000 hosts each with gathered facts. I'll try and add some more playbooks, try again and if that still doesn't show anything I'll look at testing with postgre/mysql. | 18:26 |
| ara-slack | <harlowja> hmmmm | 18:26 |
| ara-slack | <logan.attwood> @dmsimard need some DB stats or me to run some queries against my postgres? | 18:26 |
| ara-slack | <dmsimard> @harlowja btw that's without --max-requests | 18:27 |
| ara-slack | <dmsimard> @logan.attwood I've been running https://gist.github.com/dmsimard/d42cf5b281e6e6839f73756e6ab6b5a4 | 18:27 |
| ara-slack | <dmsimard> I don't happen to conveniently have access to several hundred hosts to test things on :) | 18:28 |
| ara-slack | <logan.attwood> so uh | 18:28 |
| ara-slack | <logan.attwood> try | 18:28 |
| ara-slack | <logan.attwood> 400 hosts, 2k tasks | 18:28 |
| ara-slack | <dmsimard> Is there a lot of output in those tasks ? | 18:29 |
| ara-slack | <logan.attwood> i don't believe so | 18:29 |
| ara-slack | <dmsimard> ok, I'll try. | 18:29 |
| ara-slack | <logan.attwood> but | 18:29 |
| ara-slack | <logan.attwood> we're getting around 50k tasks per run | 18:29 |
| ara-slack | <logan.attwood> (module invocations * hosts) | 18:29 |
| ara-slack | <dmsimard> I can probably reproduce with 50 tasks for each host :slightly_smiling_face: | 18:30 |
| ara-slack | <dmsimard> Just for the record, RAM usage with gunicorn/sqlite: http://paste.openstack.org/raw/718766/ from https://i.imgur.com/Clb0LUD.png | 18:36 |
| ara-slack | <dmsimard> And that's from refreshing many times until all the threads are "warmed up" | 18:36 |
| ara-slack | <dmsimard> Going to add fake tasks now.. | 18:36 |
| *** Laverne has quit IRC | 18:58 | |
| ara-slack | <dmsimard> I'm not sure exactly what I'm running... I think I'm running 1000 * 50 * 3 tasks | 19:04 |
| ara-slack | <dmsimard> It's taking forever to return lol | 19:04 |
| *** Laverne has joined #ara | 19:18 | |
| *** DrWaluigi has quit IRC | 19:36 | |
| *** DrWaluigi has joined #ara | 19:38 | |
| ara-slack | <dmsimard> @logan.attwood @harlowja okay I can definitely see a memory spike when loading a larger amount of results. Just refreshing the page yields a 2GB spike that fades as soon as it's finished loading. I'll look in that direction. | 19:45 |
| ara-slack | <harlowja> ya, +1 | 19:45 |
| ara-slack | <harlowja> something eating up memory :P | 19:45 |
| dmsimard | you guys just have too many tasks and servers :P | 19:46 |
| ara-slack | <dmsimard> Sweet, I managed to get it up to a 6GB spike | 19:55 |
| ara-slack | <dmsimard> Sorry I ever doubted you guys :( | 19:55 |
| ara-slack | <dmsimard> 6GB RAM spike from a 33MB sqlite database, how's that | 19:58 |
| dmsimard | now I'll try and run that on my laptop for easier troubleshooting. Hoping my laptop doesn't crash :D | 20:02 |
| *** tbielawa is now known as tbielawa|brb | 20:22 | |
| ara-slack | <dmsimard> @harlowja found something | 20:32 |
| ara-slack | <harlowja> :P | 20:32 |
| ara-slack | <harlowja> ya, how is it possible, lol | 20:32 |
| ara-slack | <dmsimard> https://github.com/openstack/ara/blob/master/ara/views/reports.py#L181 | 20:33 |
| ara-slack | <dmsimard> This is flushing/loading the entire sqlalchemy generator in memory | 20:33 |
| ara-slack | <dmsimard> It seems like we'd want to add yield_per to limit the size of the yields http://docs.sqlalchemy.org/en/latest/orm/query.html#sqlalchemy.orm.query.Query.yield_per | 20:34 |
| ara-slack | <harlowja> ah. cool | 20:34 |
| ara-slack | <dmsimard> If I set it to something like 100 it won't be as fast but it also won't eat all of your memory | 20:35 |
| ara-slack | <harlowja> still mind blown 6GB RAM spike from a 33MB sqlite database | 20:35 |
| ara-slack | <harlowja> lol | 20:35 |
| ara-slack | <dmsimard> what can I say | 20:35 |
| ara-slack | <dmsimard> I write efficient software | 20:35 |
| ara-slack | <harlowja> :slightly_smiling_face: | 20:35 |
| ara-slack | <dmsimard> Efficient at eating RAM of course | 20:35 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add support for configuring sqlalchemy pool size, timeout and recycle https://review.openstack.org/524427 | 20:59 |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Improve memory usage when dealing with large amount of data https://review.openstack.org/559840 | 20:59 |
| ara-slack | <dmsimard> @harlowja can you try using https://review.openstack.org/#/c/559840/ ? It include a rebase of the sqlalchemy pool patch https://review.openstack.org/524427. | 21:00 |
| dmsimard | SpamapS: ^ fyi | 21:01 |
| SpamapS | you guys are going to make me join your silly slack | 21:01 |
| dmsimard | it's cause harlowja isn't here lol | 21:02 |
| ara-slack | <harlowja> i'm here, just coding | 21:02 |
| dmsimard | ah ok, when you're coding you're on slack but not on IRC :D | 21:02 |
| dmsimard | I guess you're not... slacking ? | 21:03 |
| ara-slack | <harlowja> ya, will open IRC | 21:03 |
| ara-slack | <harlowja> ok, will now be on IRC | 21:03 |
| ara-slack | <harlowja> lol | 21:03 |
| dmsimard | whatever lol | 21:03 |
| *** harlowja has joined #ara | 21:03 | |
| harlowja | ok, now on IRC | 21:03 |
| harlowja | lol | 21:03 |
| SpamapS | is there a slack? Is it just a myth? | 21:03 |
| * SpamapS doesn't think it's ara.slack.com | 21:04 | |
| dmsimard | ara-community.slack.com | 21:04 |
| SpamapS | perhaps put that in the irc topic too :) | 21:05 |
| dmsimard | It's in the readme but I can put it here yeah | 21:05 |
| dmsimard | https://github.com/openstack/ara#contributing-testing-issues-and-bugs | 21:05 |
| *** tbielawa|brb is now known as tbielawa | 21:16 | |
| *** tbielawa has quit IRC | 21:16 | |
| *** gvincent has quit IRC | 21:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
