Tuesday, 2019-11-19

*** tosky has quit IRC00:07
*** macz has quit IRC00:08
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/train: Standardize on nginx-extras  https://review.opendev.org/69390300:15
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/train: Add possibility to overwrite public repo  https://review.opendev.org/69375700:35
*** goldyfruit has joined #openstack-ansible00:39
openstackgerritMerged openstack/openstack-ansible-os_nova master: Remove deprecated filters  https://review.opendev.org/69479800:44
*** weshay_ has joined #openstack-ansible00:48
*** redrobot has quit IRC01:07
*** gyee has quit IRC01:29
*** goldyfruit has quit IRC01:43
*** goldyfruit has joined #openstack-ansible02:41
openstackgerritMerged openstack/openstack-ansible-os_barbican master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69428003:15
*** nurdie has joined #openstack-ansible03:42
openstackgerritMerged openstack/openstack-ansible-os_tacker master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69435803:46
*** nurdie has quit IRC03:46
openstackgerritMerged openstack/openstack-ansible-os_mistral master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69432003:50
*** goldyfruit has quit IRC04:00
*** macz has joined #openstack-ansible04:04
*** macz has quit IRC04:09
*** errr has quit IRC05:16
*** pcaruana has joined #openstack-ansible05:43
*** openstackstatus has joined #openstack-ansible06:08
*** ChanServ sets mode: +v openstackstatus06:08
*** errr has joined #openstack-ansible06:46
*** mcarden has quit IRC06:49
*** mensis has joined #openstack-ansible06:52
*** mensis has quit IRC06:53
*** mensis has joined #openstack-ansible06:54
*** pcaruana has quit IRC06:56
*** udesale has joined #openstack-ansible07:06
openstackgerritinspurericzhang proposed openstack/openstack-ansible-os_swift master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69491407:28
openstackgerritinspurericzhang proposed openstack/openstack-ansible-repo_build master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69491607:30
openstackgerritinspurericzhang proposed openstack/openstack-ansible-plugins master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69491707:32
openstackgerritinspurericzhang proposed openstack/openstack-ansible-os_horizon master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69492407:46
*** cshen has joined #openstack-ansible07:54
*** ivve has joined #openstack-ansible08:17
openstackgerritShangXiao proposed openstack/openstack-ansible-ceph_client master: Fix a typo in yml file  https://review.opendev.org/69493608:24
*** tosky has joined #openstack-ansible08:26
openstackgerritShangXiao proposed openstack/openstack-ansible-os_cloudkitty master: Fix a typo in yaml file  https://review.opendev.org/69493908:30
*** pcaruana has joined #openstack-ansible08:36
*** luksky has joined #openstack-ansible08:42
openstackgerritShangXiao proposed openstack/openstack-ansible-galera_client master: Fix a type in yml file  https://review.opendev.org/69494708:49
mensisHello, i'm trying to install ELK Stack with openstack-ansible-ops but after installation elasticsearch does not work properly. After some time it throws java.lang.OutOfMemoryError and elasticsearch service fails. I was getting this error after installation but i tried to install again and right now i'm getting the error during installation , then installation fails. I'm using OpenStack stein distribution and trying to install elk_metrics_7x from m08:52
mensisaster branch of openstack-ansible-ops. Any suggestions?08:52
mensisHere is the log; http://paste.openstack.org/show/786326/08:53
*** DanyC has joined #openstack-ansible09:02
*** DanyC has joined #openstack-ansible09:02
*** DanyC has quit IRC09:03
*** DanyC has joined #openstack-ansible09:05
*** rpittau|afk is now known as rpittau09:08
jrossermensis: does it actually run out of memory? there are limits defined for the java heap size which are roughly 1/2 1/3 1/4 of the host memory size and elasticsearch will try to use it09:10
jrosserit also fails if that amount of free memory is not available09:10
jrossermensis: here is how the java heap size requirement is calculated http://codesearch.openstack.org/?q=elastic_heap_size_default09:22
*** nurdie has joined #openstack-ansible09:43
*** nurdie has quit IRC09:47
*** DanyC has quit IRC09:51
*** sshnaidm|afk is now known as sshnaidm|ruck10:10
*** DanyC has joined #openstack-ansible10:15
mensisjrosser: i've checked jvm.options file in elasticsearch_container and my heap size is 30GB, also i'm monitoring memory usage with "top" and "free -m" command, i see that i have 650GB free memory10:23
jrosserwow that is a lot of RAM. are you sure thats not including swap?10:24
mensisYes, i am. But the cpu usage of elasticsearch's java process goes between %500 and %100010:26
openstackgerritkourosh vivan proposed openstack/openstack-ansible-os_tempest master: Fix stackwiz venv pip install args  https://review.opendev.org/69497910:29
*** ansmith has joined #openstack-ansible10:32
*** ansmith_ has quit IRC10:33
jrossermensis: the cpu usage can be very large. i'd suggest disabling any beats you have installed and first get the elk/logstash to deploy cleanly and without trouble10:43
jrosseri am seeing about 10G/node steady state on mine10:44
mensisOk, thanks. I increased the index limit of metricbeat by the way and didn't solve the problem. I'll comment out the beat installation playbooks and try again11:01
*** kvivan_ has joined #openstack-ansible11:10
*** udesale has quit IRC11:17
jrosseryou can use an ansible ad-hoc command and the 'service' module to disable them all11:53
*** ansmith_ has joined #openstack-ansible12:11
*** nicolasbock has joined #openstack-ansible12:11
*** ansmith has quit IRC12:11
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server stable/train: Ansible  'search' is a test not a filter  https://review.opendev.org/69317712:15
*** mgariepy has joined #openstack-ansible12:17
*** goldyfruit has joined #openstack-ansible12:18
*** dave-mccowan has joined #openstack-ansible12:35
*** nurdie has joined #openstack-ansible12:47
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69435012:56
*** udesale has joined #openstack-ansible12:58
*** ansmith_ has quit IRC12:58
*** nurdie has quit IRC12:59
*** nurdie has joined #openstack-ansible12:59
*** nurdie has quit IRC13:03
*** chandankumar is now known as raukadah13:05
*** goldyfruit has quit IRC13:08
openstackgerritMerged openstack/openstack-ansible-os_panko master: Replace git.openstack.org with opendev.org  https://review.opendev.org/69434713:09
*** hamzy has quit IRC13:19
*** ansmith_ has joined #openstack-ansible13:44
openstackgerritkourosh vivan proposed openstack/openstack-ansible-os_tempest master: Fix stackwiz venv pip install args  https://review.opendev.org/69497913:46
taccois this a know issue that cnocchi-config is missing on the second and third node after fresh deployment with OSA?13:54
taccoi can fix this with gnocchi-upgrade but this is just a workaround for me13:54
*** luksky has quit IRC14:02
*** nurdie has joined #openstack-ansible14:07
noonedeadpunktacco: hm, personally I never heard about htat...14:10
*** DanyC has quit IRC14:10
noonedeadpunktacco: btw, just fui, gnocchi project is unmaintained now. So consider start using it wisely - ceilometer also supports at least prometheus14:14
*** goldyfruit has joined #openstack-ansible14:19
*** goldyfruit_ has joined #openstack-ansible14:28
taccook thanks for pointing this out. maybe we just throw gnocchi away. :)14:29
taccobecause we don't realy use it14:29
*** goldyfruit has quit IRC14:30
openstackgerritkourosh vivan proposed openstack/openstack-ansible-os_tempest master: Fix stackwiz venv pip install args  https://review.opendev.org/69497914:35
*** macz has joined #openstack-ansible14:36
admin0hi guys .. have you seen issues like this before: ? ERROR nova.compute.manager [instance: xxx] VirtualInterfaceCreateException: Virtual Interface creation failed14:38
admin0and i think rabbitmq is getting hammered by neutron-dhcp-agent requests14:38
*** errr has quit IRC14:40
admin0mensis, i strugged with ELK and switched over to graylog ..  i dont have good stats but the logging is solid14:41
admin0is there a config option to restrict the number of dhcp_agents per network .. for some networks, there are like 17 dhcp agents spawned14:46
cshenadmin0: dhcp_agents_per_network?14:47
admin0yep14:47
cshenit's available in neutron.conf14:47
*** DanyC has joined #openstack-ansible14:48
admin0is that equivalent to the number of network nodes ?14:49
admin0by default ?14:49
admin0its 17 in my case :D14:49
cshencheck https://docs.openstack.org/neutron/rocky/configuration/neutron.html14:49
cshenit's 1 by default14:49
admin0570 networks x 17 agents each = 9690 :D14:49
admin0its just 1 ?14:50
admin0oh .. i thought it was 2 for redundancy14:50
cshenaccording to doc, yes, it's 114:50
mgariepyone for neutron, not osa.14:50
admin0wow .. mine is : dhcp_agents_per_network = 1714:51
admin0and i did not override it14:51
mgariepyhttps://github.com/openstack/openstack-ansible-os_neutron/blob/master/templates/neutron.conf.j2#L6414:52
cshenhow redundant your network is :-D14:52
mgariepyadmin0, are you using lxb?14:52
mgariepyor ovs ?14:52
admin0ovs14:52
admin0max_l3_agents_per_router = 17 -- does that sound excessive also ?14:53
*** DanyC has quit IRC14:53
admin0can you guys do a quick check and let me know what you have in those 2 settings14:54
admin0and when I reduce the size of those to 2, i don't think neutron will kill itself the excess  15 ones right ? that i have to do myself ?14:55
cshenadmin0: for our setup, max_l3_agents_per_router = 3 and dhcp_agents_per_network = 314:58
cshenI think it's pretty common.14:58
cshenI always kill excessive neutron processes with 'kill' command.14:59
cshenif it spawns too many.14:59
admin0my 3 neutron_containers have the ips: 172.29.239.162  172.29.236.162 and 172.29.238.162 respectively ..is this a coincidence or are we making the ips similar deliberately ?15:01
cshenare the ips from the same network?15:02
admin0from the same /2215:02
cshencheck if you hard code the ips under /etc/openstack_deploy on deployer server.15:03
admin0for a fairly moderate setup ( 25 compute nodes  and bit busy) .. out of the default, is there any recommend settings to override ? like timeouts , number of workers etc in neutron and nova ?15:03
cshenfor me, the ips of neutron containers are quite random.15:03
admin0since i have to run the playbook again to overrride those values, i figure if there is more, i can add it in one go15:04
cshenI won't override the default values at the beginning if no good reason.15:04
admin0and is it recommend to increase/adjust vif_plugging_timeout and  vif_plugging_is_fatal15:04
cshenI never adjusted vif_plugging_timeout and vif_plugging_is_fatal15:05
cshendon't over optimise15:06
cjloaderjrosser: ty for +2 +W on the nginx bug15:06
admin0i guess mine is due to this 17 stuff per network .. will adjust those and see how it goes15:06
cshenadmin0: sure.15:07
*** DanyC has joined #openstack-ansible15:20
*** nurdie has quit IRC15:20
*** DanyC has quit IRC15:24
*** errr has joined #openstack-ansible15:28
admin0in user_secrets, rabbitmq_monitoring_password  .. is this used on the managemenet plugin ? if yes, what is the username ?15:28
admin0i was trying to search for the rabbit admin pass .. but  i only see cookie token and monitoring_pass15:29
noonedeadpunkadmin0: by default user is monitoring to that password15:29
*** spatel has joined #openstack-ansible15:37
*** ivve has quit IRC15:37
*** DanyC has joined #openstack-ansible15:46
*** cshen has quit IRC15:48
*** sshnaidm|ruck has quit IRC15:49
*** sshnaidm has joined #openstack-ansible15:50
*** sshnaidm is now known as sshnaidm|ruck15:50
*** pcaruana has quit IRC15:50
*** pcaruana has joined #openstack-ansible15:51
jrossernoonedeadpunk: admin0 we were playing with that today actually15:57
jrosserthe UI is only moderatley useful unless you use rabbitmqctl and apply the "administrator" tag to the monitoring user15:57
jrosserwhich is a bit surprising15:57
jrosserotherwise it won't show youall the queues15:57
noonedeadpunkyeah, with monitoring tag is show not everything and just for reading15:58
jrosserthe default is to only show the queues that belong to the user, and in our setup thats none15:58
noonedeadpunkI guess it should be able to read overall stats?15:58
admin0i was able to reverse sort by queue and figure out that its the 3 neutron server containers where most of the messageed were queued .. when i check the neutron logs, its working as normal .. but the process is at 100% in the cpu15:58
admin0so either i need more neutron server containers .. or i need faster cpus15:59
noonedeadpunkI think I've created admin with my own15:59
jrossernoonedeadpunk: with the standard setup it was useless for looking at the overall rabbitmq status15:59
jrosserbecause the monitoring user didnt have visibility of anything helpful, not even readonly15:59
noonedeadpunkhm... that's strange.. I guess I'm using user just with monitoring tag for checking cluster status and number of messages depending on type16:00
*** udesale has quit IRC16:00
noonedeadpunks/I am/I was/16:00
noonedeadpunkbtw16:01
noonedeadpunk#startmeeting openstack_ansible_meeting16:01
openstackMeeting started Tue Nov 19 16:01:07 2019 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:01
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:01
noonedeadpunk#topic office hours16:01
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:01
noonedeadpunkSo, Queens has moved to extended maintenance status https://review.opendev.org/#/c/691318/16:01
noonedeadpunkI guess that evrardjp was going to discuss this here, but it's already merged, so...16:02
guilhermespo/16:03
noonedeadpunkAlso I think we should wait for a while for the next rc until upgrade bugs will be figured out16:04
*** gyee has joined #openstack-ansible16:05
noonedeadpunkAnd we have kinda problem with upgrade jobs... So first of all they're pretty close to timouts. For telemetry they fails with timeout for centos.16:06
noonedeadpunkAnother thing is that we can't use depends-on for lower release upgrade jobs. And as it's not in the independent template - we can't set them as non-voting or modify/exclude per role.16:09
noonedeadpunkSo for horizon train they're going to fail, as stein don't allow doing metal horizon deployments16:09
cjloadero/16:10
cjloadernoonedeadpunk: how much longer do I have to complete inspector to get it in the next rc for train?16:11
jrosserdo we use bootstrap-ansible.sh to get both releases in place for upgrade jobs?16:12
jrosseri'm not sure depends-on will work for either branch16:12
noonedeadpunkyes, we do. Actually run-upgrade.sh is supposed to. But, we're checking out back to the head...16:16
noonedeadpunkBut yes, not sure either...16:16
noonedeadpunkAnd it looks like a problem, especially when it's inlcuded to all roles at once.16:16
noonedeadpunkOn the other hand - they already helped us several times and prevented from breaking upgrade16:16
noonedeadpunkSo they are pretty useful.16:17
noonedeadpunkcjloader: so we have to release no later than 15 of december16:17
cjloaderokay16:18
cjloaderi'll keep you guys informed16:18
jrossernoonedeadpunk: also have you seen a bunch of broken things with changing tempest to run smoke test?16:20
noonedeadpunkYep, I did:(16:21
jrosserit is good and bad16:22
jrosserlots of broken <- bad16:22
jrosserlots more test coverage <- good16:22
noonedeadpunktbh I never was good enough in fixing stuff to apply full tempest test...16:22
jrosserbut biggest problem seems to be the ceph jobs which are broken now on master16:22
noonedeadpunkoh, I haven't realized that16:23
jrosserlike this https://review.opendev.org/#/c/694253/16:25
jrosseri *think* this is related to the tempest test being more thorough16:25
noonedeadpunkaccording to https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eac/694253/2/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/eace31a/logs/openstack/aio1_utility_container-1101bcb3/utility/stestr_results.html it fails swift...16:27
noonedeadpunkbut the thing is that swift is generally fails https://review.opendev.org/#/c/694783/16:28
noonedeadpunkso not sure that problem is in ceph itself16:28
noonedeadpunkthe patch don't fix stuff for ubuntu distro now, but it shows that actually nothing pass tests16:30
jrosserfor swift it is this https://zuul.opendev.org/t/openstack/build/4a20e01c392e4e2d93f228de2ff9e30e/log/logs/openstack/aio1-utility/tempest_run.log.txt.gz#17516:31
jrosseri think horizon needs tempest telling to ignore the certificate check for public endpoint16:31
jrosserand then congress is broken, looks like a bunch of missing config in the service16:32
noonedeadpunkyep. And it looks like we don't deploy reseller user16:32
noonedeadpunkthis feels like tons of work to clean everythin out16:33
noonedeadpunkbut tbh I'm not sure we have an experts for each supported project16:34
jrosseri think 4-6 roles are affected badly16:34
noonedeadpunkI'd say swift is a critical one16:35
jrosserimho improving test coverage is a good aim for this cycle, if we don't already have one16:35
jrosserbut right now when we try to release, not so good16:35
noonedeadpunk++16:36
noonedeadpunkofc we can set some of stuff to non-voting untill release... But not sure it's the path to follow16:36
jrosseror we can set this var back to how it was https://github.com/openstack/openstack-ansible-os_tempest/commit/6116c681a4119aed9b1fa7723de18ae0731ff500#diff-7eeda618087b49ae876084ab6c73fdbbL85-L9916:37
jrosserin the integrated repo16:37
jrosserand leave the smoke test patch in os_tempest as an indicator of where we want to end up16:37
noonedeadpunkthat's nice idea16:37
noonedeadpunkand rollback this right after release16:38
jrosseryes, or make a pile of experimental jobs on the things we know break16:38
* noonedeadpunk always forget about experimental jobs16:39
jrosserperhaps too we should add more scenraios on the os_tempest repo, becasue thats really whats caught is out here16:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Roll back use of tempest smoke test for the integrated repo  https://review.opendev.org/69503316:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_designate master: Remove deprecated packages from centos installs  https://review.opendev.org/69477516:49
jrosserlets see if that works ^16:49
admin0anyone seen this on console and a jumbled display ? Unimplemented function 108(Inval All Palettes) [ further notices suppressed ]16:57
admin0Unimplemented function 102(Display Mark) [ further notices suppressed ]16:57
noonedeadpunk#endmeeting17:00
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"17:00
openstackMeeting ended Tue Nov 19 17:00:20 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-11-19-16.01.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-11-19-16.01.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-11-19-16.01.log.html17:00
*** rpittau is now known as rpittau|afk17:06
*** ivve has joined #openstack-ansible17:21
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_nova master: Disable nova-spicehtml5proxy service when installing novnc  https://review.opendev.org/69504517:26
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_nova master: Disable spicehtml5proxy and serialproxy when installing novnc  https://review.opendev.org/69504518:22
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_nova master: Disable other remote access methods when installing nova-spicehtml5proxy  https://review.opendev.org/69505918:27
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_nova master: Disable spicehtml5proxy and serialproxy when installing novnc  https://review.opendev.org/69504518:28
*** kvivan_ has quit IRC18:28
*** nicolasbock has quit IRC18:34
*** nicolasbock has joined #openstack-ansible18:34
*** DanyC has quit IRC18:41
jrossernoonedeadpunk: looks like https://review.opendev.org/#/c/694775/ is going to pass, the tempest whitelist rollback looks OK18:51
*** nicolasbock has quit IRC19:14
*** nicolasbock has joined #openstack-ansible19:14
admin0what could be a reason where half of the instances have proper console and half of them have garbage prompts19:17
mgariepyper host?19:25
*** goldyfruit___ has joined #openstack-ansible19:55
*** goldyfruit_ has quit IRC19:57
*** tosky has quit IRC20:27
*** openstackgerrit has quit IRC20:35
admin0is there an average number of  (ready) messages you see in rabbitmq queue (0 in unack) and beyond that needs some check ?20:43
*** mgariepy has quit IRC20:53
*** openstackgerrit has joined #openstack-ansible20:54
openstackgerritLance Bragstad proposed openstack/openstack-ansible-os_keystone master: Remove references to writable LDAP from documentation  https://review.opendev.org/69508120:54
admin0checking if you guys know how to check the ready messages is and where its stuck20:57
*** goldyfruit_ has joined #openstack-ansible20:58
*** goldyfruit___ has quit IRC21:00
openstackgerritBjoern Teipel proposed openstack/openstack-ansible master: Disable journald-remote playbook  https://review.opendev.org/69508321:07
openstackgerritBjoern Teipel proposed openstack/openstack-ansible master: Disable journald-remote playbook  https://review.opendev.org/69508321:10
*** nicolasbock has quit IRC21:20
*** mcarden has joined #openstack-ansible21:28
*** nicolasbock has joined #openstack-ansible21:35
*** spatel has quit IRC21:39
*** ansmith_ has quit IRC21:42
*** andrea15 has quit IRC21:46
*** andrea15 has joined #openstack-ansible21:47
*** andrea15 has quit IRC21:48
*** andrea15 has joined #openstack-ansible21:48
*** andrea15 has quit IRC21:49
*** andrea15 has joined #openstack-ansible21:49
*** andrea15 has quit IRC21:50
*** andrea15 has joined #openstack-ansible21:51
*** andrea15 has quit IRC21:52
*** andrea15 has joined #openstack-ansible21:52
jrosserreviews needed... this unblocks master https://review.opendev.org/#/c/69503321:56
*** DanyC has joined #openstack-ansible22:28
*** pcaruana has quit IRC22:33
*** macz has quit IRC22:35
*** tosky has joined #openstack-ansible23:24
*** tosky has quit IRC23:26
*** ansmith_ has joined #openstack-ansible23:34
*** ivve has quit IRC23:54
*** goldyfruit_ has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!