16:00:24 <noonedeadpunk> #startmeeting openstack_ansible_meeting
16:00:25 <openstack> Meeting started Tue Nov  3 16:00:24 2020 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:28 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:00:43 <noonedeadpunk> #topic rollcall
16:00:47 <noonedeadpunk> o/
16:00:55 <mgariepy> hey !
16:04:10 <noonedeadpunk> seems we won't have much more attandees :)
16:04:32 <noonedeadpunk> s/:)/:(/
16:04:47 <noonedeadpunk> #topic bug triage
16:05:38 <mgariepy> for once i'm here.
16:06:04 <noonedeadpunk> yeah:)
16:06:11 <jamesdenton> o/
16:06:18 <gshippey> o/
16:06:36 <noonedeadpunk> so we have one new bug and it jamesdenton one
16:06:40 <noonedeadpunk> https://bugs.launchpad.net/openstack-ansible/+bug/1902585
16:06:41 <openstack> Launchpad bug 1902585 in openstack-ansible "uwsgi socket backlog exceeded" [Undecided,New]
16:07:05 <jamesdenton> ahh yes
16:08:35 <noonedeadpunk> well I think we need to bump limits here indeed
16:08:52 <noonedeadpunk> not sure about reasonable defaults part)
16:09:01 <noonedeadpunk> as 1024 sounds as enough...
16:09:51 <jamesdenton> i think there is a memory penalty.. maybe 8kB per, or something like that. nothing huge
16:10:07 <noonedeadpunk> jamesdenton: `some buffer` means you've bumped smth else except mentioned things?
16:10:29 <jamesdenton> well yes, we did 1024 and it worked for a bit, but then that limit was hit. and so on
16:10:43 <jamesdenton> i'm not really sure why the queue wasn't emptied fast enough
16:10:56 <mgariepy> how many hosts ?
16:11:19 <noonedeadpunk> I think it was about how many connections it was serving
16:12:10 <jamesdenton> good question, i am not sure. looking
16:14:17 <jamesdenton> 137 computes
16:23:26 <noonedeadpunk> sorry got distracted
16:23:34 <noonedeadpunk> so, um
16:24:05 <noonedeadpunk> eventually it's pretty easy to override uwsgi listen
16:25:24 <jamesdenton> true.
16:25:29 <noonedeadpunk> but not globally....
16:26:20 <noonedeadpunk> the one thing I was facing rcently is need to bump prlimit
16:26:47 <noonedeadpunk> * LimitNOFILE
16:26:56 <noonedeadpunk> but it feels it's not the case here
16:27:11 <noonedeadpunk> So what I'm trying to understand if we should change defaults and how exactly
16:27:57 <jamesdenton> right, i am not sure. it might help to have an override available and maybe a directive on how to use it. but keep the defaults as-is? They seem to work OK except in the case of a severely downsized control plane
16:28:10 <jamesdenton> in out case we went from 3->1
16:28:14 <jamesdenton> *our
16:28:37 <jamesdenton> and i don't have data at this time to show normal load vs connections growing at that time
16:33:10 <noonedeadpunk> well, sysctl can be overwritten with openstack_user_kernel_options
16:33:18 <noonedeadpunk> https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L129
16:33:28 <noonedeadpunk> and uwsgi can also be overwritten...
16:33:46 <noonedeadpunk> the only thing missing is global override for all uwsgi services
16:34:05 <noonedeadpunk> will this cover the usecase?
16:37:59 <noonedeadpunk> however this downsize is kind of example of the upgrade scenario
16:39:15 <jamesdenton> yeah, the kernel param is no issue. but would the listen override need to go here? https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2
16:39:28 <jamesdenton> and that template is used for individual uwsgi apps, right?
16:39:38 <noonedeadpunk> yep
16:39:46 <noonedeadpunk> and I think each service has overrides
16:40:11 <noonedeadpunk> like nova_api_metadata_uwsgi_ini_overrides
16:40:17 <jamesdenton> ahh ok
16:40:32 <jamesdenton> yes i think you're right
16:41:15 <spatel> jamesdenton: I have now DPDK supported OVS
16:41:37 <jamesdenton> nova_api_os_compute_uwsgi_ini_overrides in this case. ok, so maybe just a docs change?
16:41:59 <jamesdenton> looks like somaxconn is bumped to 4096 in latest 5.x kernel
16:42:17 <noonedeadpunk> yeah, I guess we can just document things
16:42:43 <jamesdenton> of course, folks have to READ :D
16:42:56 <noonedeadpunk> and I will add patch to uwsgi to be able to have overwrite for all services at once rather than do it for each separately
16:43:03 <jamesdenton> cool
16:45:53 <noonedeadpunk> btw, where do you see docs?
16:46:17 <noonedeadpunk> should we do it for each role or just some small section in the integrated repo?
16:46:22 <jamesdenton> hmm
16:47:17 <jamesdenton> https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-advanced-config-override.html
16:47:18 <jamesdenton> ?
16:47:29 <jamesdenton> seems old
16:48:01 <noonedeadpunk> what is draft....
16:48:33 <jamesdenton> no idea
16:48:39 <noonedeadpunk> lol what is it:)
16:49:03 <noonedeadpunk> well, I'm not sure we can maintain all type of overrides...
16:49:55 <noonedeadpunk> I'd rather drop list of `Currently available overrides` and did describe general approach or naming convention we're using and asking to double check naming for each role
16:50:03 <jamesdenton> agreed
16:50:16 <noonedeadpunk> well, we already have such note....
16:50:29 <noonedeadpunk> but yeah. we need to fix this page
16:51:31 <noonedeadpunk> #agreed document way of overrides usage (including uwsgi ones). use https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-advanced-config-override.html as base
16:55:22 <openstackgerrit> Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-uwsgi master: Allow to globaly override uwsgi params  https://review.opendev.org/761198
16:55:42 <noonedeadpunk> #topic office hours
16:56:23 <noonedeadpunk> eventually what I wanted to say is that next wednesday stein goes to extended maintenance, so I'm going to do last release this week
16:56:52 <noonedeadpunk> and afterwards I will switch branches to stable/stein for services and roles, to tag it with EM
16:57:19 <noonedeadpunk> so that there was no need in bumps (as they won't have much sense anyway)
16:57:32 <noonedeadpunk> and thanks everyone for attending!
16:57:42 <noonedeadpunk> #endmeeting