Friday, 2018-10-26

ThiagoCMCHey guys, the following command/"TASK" is failing when with nspawn: "wget http://172.29.239.250:8181/os-releases/18.0.0/ubuntu-18.04-x86_64/requirements_absolute_requirements.txt", error: 503: Service Unavailable.00:02
ThiagoCMCAny idea?00:02
*** aredinger has joined #openstack-ansible00:03
ThiagoCMCTASK [Get list of repo packages]00:05
ThiagoCMCUtility container00:05
ThiagoCMCOnly 1 of 3 failed! I don't think that it is a coincidence that it failed right on the same machine where the VIP is!  Hmmm...00:06
*** aredinger has quit IRC00:07
ThiagoCMCI'll ignore this bug, since I have 2 more utility containers =P00:07
*** marst has quit IRC00:21
*** mmercer has quit IRC00:37
*** cshen has joined #openstack-ansible00:51
*** fatdragon128 has joined #openstack-ansible00:56
*** cshen has quit IRC00:56
*** fatdragon has quit IRC00:56
*** nsmeds_ has quit IRC00:57
*** fatdragon128 has quit IRC00:58
*** ansmith_ has joined #openstack-ansible01:02
*** aredinger has joined #openstack-ansible01:03
*** aredinger has quit IRC01:08
*** lbragstad has quit IRC01:49
*** lbragstad has joined #openstack-ansible01:49
*** aredinger has joined #openstack-ansible02:03
*** aredinger has quit IRC02:10
*** adrianreza has joined #openstack-ansible02:31
*** aredinger has joined #openstack-ansible02:35
*** aredinger has quit IRC02:51
*** cshen has joined #openstack-ansible02:52
*** cshen has quit IRC02:57
*** aredinger has joined #openstack-ansible03:03
*** aredinger has quit IRC03:07
*** aredinger has joined #openstack-ansible03:22
*** aredinger has quit IRC03:27
*** udesale has joined #openstack-ansible03:51
*** aredinger has joined #openstack-ansible03:59
*** aredinger has quit IRC04:07
*** dave-mccowan has quit IRC04:14
*** ThiagoCMC has quit IRC04:31
*** aredinger has joined #openstack-ansible05:03
*** aredinger has quit IRC05:07
*** shyamb has joined #openstack-ansible05:25
*** shyamb has quit IRC05:37
*** shyamb has joined #openstack-ansible05:47
*** cshen has joined #openstack-ansible05:50
*** dw_ has joined #openstack-ansible05:50
*** evrardjp_ has joined #openstack-ansible05:51
*** irclogbot_0 has quit IRC05:55
*** evrardjp has quit IRC05:55
*** mattoliverau has quit IRC05:55
*** bradm has quit IRC05:55
*** chkumar|off has quit IRC05:55
*** arbrandes1 has quit IRC05:55
*** mpjetta has quit IRC05:55
*** dw has quit IRC05:55
*** arbrandes1 has joined #openstack-ansible06:02
*** aredinger has joined #openstack-ansible06:03
*** chandankumar has joined #openstack-ansible06:06
*** aredinger has quit IRC06:07
*** shyamb has quit IRC06:13
*** faizy_ has joined #openstack-ansible06:20
*** faizy98 has quit IRC06:24
*** cshen has quit IRC06:28
*** shyamb has joined #openstack-ansible06:31
*** cshen has joined #openstack-ansible06:53
*** aredinger has joined #openstack-ansible07:03
*** aredinger has quit IRC07:07
*** DanyC has joined #openstack-ansible07:18
*** shardy has joined #openstack-ansible07:26
*** shyamb has quit IRC07:28
evrardjp_prometheanfire: read two lines above :)07:37
*** evrardjp_ is now known as evrardjp07:37
*** shyamb has joined #openstack-ansible07:39
*** vnogin has joined #openstack-ansible07:44
*** tosky has joined #openstack-ansible07:46
*** vnogin has quit IRC07:49
*** DanyC has quit IRC07:53
openstackgerritKimMinsik proposed openstack/openstack-ansible master: Aodh group vars has wrong admin and internal endpoint url  https://review.openstack.org/61352107:54
*** thuydang has joined #openstack-ansible07:55
*** aredinger has joined #openstack-ansible08:03
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142408:03
chandankumarevrardjp: done :-)08:03
*** hamzaachi has joined #openstack-ansible08:03
*** shyamb has quit IRC08:03
*** thuydang has quit IRC08:07
*** aredinger has quit IRC08:07
*** thuydang has joined #openstack-ansible08:07
*** vollman has quit IRC08:11
*** DanyC has joined #openstack-ansible08:16
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142408:18
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142408:19
*** thuydang has quit IRC08:23
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Use consistent apt options for upgrades  https://review.openstack.org/61340708:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Use consistent apt options for upgrades  https://review.openstack.org/61352608:28
odyssey4mecores, thanks to some upstream image shenanigans for ubuntu we'll need to get https://review.openstack.org/613526 into stable/rocky asap08:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Use consistent apt options for upgrades  https://review.openstack.org/61352708:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/pike: Use consistent apt options for upgrades  https://review.openstack.org/61352808:31
odyssey4mewell, make that https://review.openstack.org/#/q/Ic6689387a28539b7eb341d55a9db08f078e8e975 jrosser evrardjp hwoarang08:33
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: [DNM] - Testing OBS Master  https://review.openstack.org/57054308:33
mbuilit would be great if somebody could review: https://review.openstack.org/#/c/613104/, thanks :)08:33
hwoarangchecking08:33
hwoarangodyssey4me: no problem it's not a huge change08:34
*** cshen has quit IRC08:35
hwoarangi wish i knew why leap15 is so much slower in zuul compared to 42.3 :/08:36
eumel8odyssey4me: now we're ready for you: https://review.openstack.org/#/c/612510/08:38
eumel8unfortunatelly the translated index pages will not shown yet. The fix is hopefully in https://review.openstack.org/#/c/613531/08:40
odyssey4meeumel8: ah, awesome :)08:41
odyssey4meeumel8: does that doc job do doc8, or do we still need to do doc8 in our lint test?08:41
eumel8odyssey4me: it seems so: http://logs.openstack.org/10/612510/2/check/openstack-tox-docs/49b2fec/job-output.txt.gz#_2018-10-26_06_38_00_93768108:44
odyssey4meeumel8: ah, that's because we told it to: https://github.com/openstack/openstack-ansible/blob/master/tox.ini#L4508:51
*** spatel has joined #openstack-ansible08:52
*** spatel has quit IRC08:56
*** cshen has joined #openstack-ansible08:59
*** hamzaachi has quit IRC09:01
*** aredinger has joined #openstack-ansible09:03
*** shyamb has joined #openstack-ansible09:05
*** aredinger has quit IRC09:08
eumel8odyssey4me: ah, ok, got it. What I saw, the Zuul job simply calls tox -edocs and there is no doc8 extra. So it should be in tox.ini09:12
odyssey4meeumel8: yeah, so we're good on that front for now09:17
eumel8:odyssey4me: yeah, I have one trivial proposal for doc fixes: https://review.openstack.org/#/c/602866/ Later we need a deeper look into build warnings. There are some more.09:20
evrardjpeumel8: thanks!09:32
openstackgerritDamian Cikowski proposed openstack/openstack-ansible-os_cinder master: tasks: cinder_qos: Use the new OpenStack CLI client  https://review.openstack.org/56121809:43
*** shyamb has quit IRC09:43
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-openstack_hosts master: SUSE: Add system-user-nobody for Leap 15  https://review.openstack.org/61354709:55
*** aredinger has joined #openstack-ansible09:55
*** electrofelix has joined #openstack-ansible09:58
*** aredinger has quit IRC09:59
*** aredinger has joined #openstack-ansible10:03
*** hamzaachi has joined #openstack-ansible10:03
evrardjpthanks hwoarang10:06
openstackgerritMerged openstack/openstack-ansible master: [trivial] docs: fix sphinx warnings unknown options  https://review.openstack.org/60286610:06
*** aredinger has quit IRC10:07
*** shyamb has joined #openstack-ansible10:12
*** shyamb has quit IRC10:24
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/rocky: Use consistent apt options for upgrades  https://review.openstack.org/61352610:34
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/pike: Use consistent apt options for upgrades  https://review.openstack.org/61352810:54
openstackgerritMerged openstack/openstack-ansible master: Example setup for gnocchi with ceph and redis  https://review.openstack.org/58294610:55
*** sum12 has quit IRC11:00
openstackgerritMerged openstack/openstack-ansible-memcached_server stable/rocky: Change netcat to be a required package  https://review.openstack.org/61329011:00
openstackgerritMerged openstack/openstack-ansible-memcached_server stable/ocata: Change netcat to be a required package  https://review.openstack.org/61329311:00
openstackgerritMerged openstack/openstack-ansible-memcached_server stable/queens: Change netcat to be a required package  https://review.openstack.org/61329111:00
openstackgerritMerged openstack/openstack-ansible-memcached_server stable/pike: Change netcat to be a required package  https://review.openstack.org/61329211:00
*** sum12 has joined #openstack-ansible11:02
*** aredinger has joined #openstack-ansible11:03
*** aredinger has quit IRC11:07
*** vnogin has joined #openstack-ansible11:08
*** dave-mccowan has joined #openstack-ansible11:14
*** shyamb has joined #openstack-ansible11:15
*** tridde is now known as trident11:17
*** vnogin has quit IRC11:25
*** udesale has quit IRC11:28
*** vnogin has joined #openstack-ansible11:32
*** hamzaachi has quit IRC11:35
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update lxc_hosts role SHA for apt failures  https://review.openstack.org/61356011:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update repo_build role SHA for venv build fail fix  https://review.openstack.org/61332711:37
odyssey4meevrardjp: FYI https://review.openstack.org/613560 and https://review.openstack.org/613327 must merge before we request a release for rocky, they're both release blockers11:38
*** ansmith_ has quit IRC11:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update lxc_hosts role SHA for apt failures  https://review.openstack.org/61356111:40
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/queens: Use keystone_auth for credentials  https://review.openstack.org/61219311:45
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/queens: Disable sending metrics for clusters  https://review.openstack.org/61219211:47
*** strobelight has joined #openstack-ansible11:48
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: SUSE: Fix version compare for OBS containers repository  https://review.openstack.org/61333412:00
*** aredinger has joined #openstack-ansible12:03
*** aredinger has quit IRC12:07
*** shyamb has quit IRC12:12
*** strattao has joined #openstack-ansible12:16
*** vnogin has quit IRC12:17
*** strobelight_ has joined #openstack-ansible12:29
*** strobelight is now known as Guest8535312:29
*** strobelight_ is now known as strobelight12:30
*** Guest85353 has quit IRC12:32
*** shyamb has joined #openstack-ansible12:34
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/ocata: Add memcache flushing handler on db migrations  https://review.openstack.org/61332512:45
*** hamzaachi has joined #openstack-ansible12:46
*** shyamb has quit IRC12:47
*** aredinger has joined #openstack-ansible12:47
*** ansmith_ has joined #openstack-ansible12:59
*** dave-mccowan has quit IRC13:00
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-log-collect: Improve rsync performance for log collection  https://review.openstack.org/61357313:01
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/pike: Add memcache flushing handler on db migrations  https://review.openstack.org/61332313:06
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Clean up a-r-r role for upgrade test removal  https://review.openstack.org/61333113:08
chandankumarodyssey4me: mnaser https://review.openstack.org/#/c/591424/ distro support tempest patch is good to go13:10
chandankumarodyssey4me: evrardjp thank you for all the help on this patch :-)13:11
odyssey4meLGTM, thanks chandankumar :)13:11
chandankumar\o/13:12
chandankumarthanks :-)13:12
chandankumarodyssey4me: I think we can other projects use tempest plugins rpm and to run tests what you say?13:14
chandankumar*can enable13:14
odyssey4mechandankumar: by 'other projects' what do you mean?13:14
chandankumarodyssey4me: I mean running cinder specific tempests if cinder is configured using rpm then running cinder tempest plugins tests from python-cinder-tests-tempest rpm package in ci job13:15
chandankumaragaisnt os_cinder project13:15
odyssey4mechandankumar: we already use the source builds for that in all OSA roles13:15
odyssey4mechandankumar: also, once that merges - https://github.com/openstack/openstack-ansible-tests/blob/master/test-distro_install-vars.yml#L54 is already there, so all distro install tests will start to use the distro install instead immediately13:17
odyssey4mechandankumar: I hope you're ready to get patching when things start breaking ;)13:17
chandankumarodyssey4me: but we need to add the right package name for the same for example here https://github.com/openstack/openstack-ansible-os_cinder/blob/master/vars/redhat-7.yml13:18
*** spatel has joined #openstack-ansible13:19
chandankumarAh it is removed from here https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tests/os_cinder-overrides.yml13:19
chandankumarodyssey4me: I can patch if things will break13:19
odyssey4mechandankumar: so jrosser and I were discussing something around this yesterday13:20
odyssey4mecurrently we have a bunch of the plugin repositories set here: https://github.com/openstack/openstack-ansible-os_tempest/blob/master/vars/main.yml13:20
odyssey4meI suspect that we might need to revise things so that we have an easy way to enable installing plugins for services in the os_tempest role based on whatever's enabled13:22
odyssey4methis will need to be done for source and distro builds13:22
chandankumarodyssey4me: yup agree13:22
openstackgerritMerged openstack/openstack-ansible-tests master: Clean up unused upgrade test tooling  https://review.openstack.org/61333213:23
odyssey4mehttps://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L149-L160 is there and could be extended and leveraged, although that's not particularly automatic13:23
odyssey4meperhaps extending that and using them is a good start13:24
odyssey4methen we can make it more automatic later13:24
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60945013:24
chandankumarodyssey4me: yes I will add to the list13:25
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60896713:25
odyssey4mecentralising everything into the tempest role makes it simpler to maintain - we'd be able to remove the plugin setting and test whitelist from all the roles and just let the tempest role figure it out13:25
openstackgerritMerged openstack/openstack-ansible-repo_build stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871413:25
openstackgerritMerged openstack/openstack-ansible-os_aodh stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871513:26
openstackgerritMerged openstack/openstack-ansible-os_zaqar stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60944713:26
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/queens: Use consistent apt options for upgrades  https://review.openstack.org/61352713:27
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60896613:27
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871313:28
openstackgerritMerged openstack/openstack-ansible-os_nova master: Use distro_install project-template  https://review.openstack.org/60843113:28
openstackgerritMerged openstack/openstack-ansible-rsyslog_server stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60896213:28
openstackgerritMerged openstack/openstack-ansible-os_ceilometer stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871213:30
openstackgerritMerged openstack/openstack-ansible-rsyslog_client stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871013:30
openstackgerritMerged openstack/openstack-ansible-tests master: Add bionic py3 test and switch linters/integrated to bionic  https://review.openstack.org/61267213:30
openstackgerritMerged openstack/openstack-ansible-pip_install stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871113:30
openstackgerritMerged openstack/openstack-ansible-os_rally stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60896513:30
openstackgerritMerged openstack/openstack-ansible-haproxy_server stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60871613:31
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/rocky: Replace Chinese punctuation with English punctuation  https://review.openstack.org/60870813:31
*** irclogbot_0 has joined #openstack-ansible13:31
odyssey4mehwoarang: if I read this right, the ansible_distribution_family for all versions is 'Suse', right?13:32
hwoarangyes13:33
*** chandankumar is now known as chkumar|off13:33
odyssey4mehwoarang: thanks13:34
odyssey4mehrm, that should've been ansible_os_family13:35
mnasersome love to https://review.openstack.org/#/c/612678/2 would be nice :>13:36
mnaserconsidering the role is broken for anything beyond an aio13:37
*** aredinger has quit IRC13:38
*** spatel has quit IRC13:51
*** aredinger has joined #openstack-ansible13:52
mnaserodyssey4me: are you doing work on the os_octavia role these days?13:56
*** DanyC_ has joined #openstack-ansible13:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add default distro packages for wheel builds  https://review.openstack.org/61358513:58
odyssey4memnaser: I help out from time to time - mainly for things I broke.13:58
mnaserodyssey4me: well you just broke something and im pushing up a patch :P14:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add default distro packages for wheel builds  https://review.openstack.org/61358514:01
odyssey4memnaser: oh dear, I'm sorry :/14:01
*** DanyC has quit IRC14:01
odyssey4meBreaking things is a sign of progress though. ;)14:01
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_octavia master: Use octavia_service_setup_host for getting facts  https://review.openstack.org/61358714:01
mnaserodyssey4me: aha, no worries, it was a good fix!14:01
mnaserodyssey4me: ^14:01
mnaseri didnt test that yet but seeing no delegation here and a "shade doesn't exist!!1!" error here14:02
odyssey4mejrosser: so, as we discussed yesterday, there's definitely a need to cover at least a base set of packages for *all* roles when building wheels or installing directly from pip - https://review.openstack.org/613585 aims to cover that requirement14:02
odyssey4memnaser: hmm, on which task? it's supposed to either use the ansible runtime venv or the octavia venv14:02
mnaserin https://review.openstack.org/61358714:02
mnaserit tries to get the facts which before it used to do in the mgmt and flavor task14:03
mnaserwhich did it in the right venv or whatever but moved out the block which meant it went back to going to the server that didnt have shade14:03
odyssey4memnaser: look at line 92-9314:03
odyssey4meI suspect you forgot to rebuild your repo, and thus shade is missing in the octavia venv14:03
mnaseroh it was readded?14:04
mnaseris there any reason why we didnt want to maintain/run that locally?14:04
mnasersimiliar to all the others?14:04
odyssey4memnaser: https://github.com/openstack/openstack-ansible-os_octavia/commit/e0a69d0bda3f55fdaf1e4cdb40baffe3a638d20114:04
mnaseroh14:04
mnaseri guess set_fact gets delegated too right14:04
odyssey4mefor some reason when using delegation the facts weren't on the target hosts14:04
mnaserso we end up set_fact locally14:04
mnaserah ok14:05
mnaserso you DIDNT break anything14:05
odyssey4meoriginally I tried using localhost to get the facts, then use those but it didn't seem to work so I switched to adding shade to the venv instead14:05
mnaserabandoned with an appropriate message14:05
odyssey4meI was kinda under time pressure, so if there is a better way, I'm all ears - but that works as-is.14:05
odyssey4me(I verified using a multi-node env)14:06
mnaseri'm under the same so i can look into digging into the depths of the why some other time14:06
mnaseryeah your fix makes sense14:06
mnaseri didnt see the add shade to venv part14:06
odyssey4mehwoarang evrardjp cloudnull I'd like your reviews for https://review.openstack.org/613585 too please. It's a key part of getting rid of repo-build for good.14:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update lxc_hosts role SHA for apt failures  https://review.openstack.org/61359414:11
*** dave-mccowan has joined #openstack-ansible14:11
*** spatel has joined #openstack-ansible14:14
*** luksky has joined #openstack-ansible14:14
*** dave-mccowan has quit IRC14:15
*** vrobert has joined #openstack-ansible14:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-config_template master: Switch python3 test to bionic  https://review.openstack.org/61359514:18
*** spatel has quit IRC14:18
vroberthello developers14:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Switch python3 test to bionic and add it to gate  https://review.openstack.org/61359614:20
vrobertI have a problem with ansible-hardening role, eating up all my 11GB memory because of dynamic includes in tasks/rhel7stig/main.yml14:22
*** cshen has quit IRC14:23
odyssey4mevrobert: that sounds a bit odd, given that we do all testing on 8GB RAM hosts14:23
vrobertwhy are you using dynamic includes here: - include_tasks: accounts.yml - include_tasks: aide.yml - include_tasks: auditd.yml - include_tasks: auth.yml ...14:23
vrobert?14:23
vrobertthere are no loops or tags or vars around these lines14:24
*** cshen has joined #openstack-ansible14:24
*** ThiagoCMC has joined #openstack-ansible14:24
odyssey4mevrobert: if there are no conditionals, then those can be converted into import_tasks instead - I think the person who did the patch didn't know the difference, so a patch would be welcome to fix that14:24
vrobertwhen I changed to import_tasks the mem usage stayed 1.5GB14:24
odyssey4meare you able to push up a patch for that?14:25
*** tosky has quit IRC14:25
vrobertSeems major convert includes wrong in this commit: https://github.com/openstack/ansible-hardening/commit/74c904247fb8ad69848db41408df0c812826915d14:25
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142414:26
vrobertat ansible 2.1 there was only include statement14:26
*** vnogin has joined #openstack-ansible14:26
odyssey4meI didn't realise the impact was that heavy. I think that means we're going to have to fix that across all OSA repositories.14:26
vrobertyes. I am using osa 17.x.x branch14:27
vrobertwhich is using ansible 2.4.4.014:27
vrobertand I think this version affected these kinds of memory bug or mem balloning problem14:27
noonedeadpunkI've stuck with rabbitmq... Once I do "systemctl stop/start rabbitmq-server" servers, openstack services gut stuck, and just do nothing. For example, nova-compute doesn't see conductor, while it's running (but probably don't recieve messages by rabbit). Once I run rabbitmq-setup.yml -e rabbitmq_upgrade=true - problem dissapears.14:27
vrobertyou can check it here: https://github.com/ansible/ansible/issues/3044114:27
noonedeadpunkAnd I don't fully understand the reason of this and what exactly fixes the problem14:28
*** tosky has joined #openstack-ansible14:29
odyssey4mevrobert: could you register a bug for it? we'd need to do a bunch of patches across a bunch of repositories, and backport them - so a bug reference would act as a good way to keep track of all of the changes14:30
logan-vrobert: has a fix been backported to 2.4? maybe we can bump the 2.4 minor version to apply the fix14:30
vrobertI tried to hack the bootstrap-ansible.sh script to use the latest 2.4.6 but didn't help.14:31
*** vollman has joined #openstack-ansible14:32
vrobertThe only thing that helped for me when I changed the include_tasks to import_tasks.14:33
logan-are you able to determine if this occurs on more recent ansible versions we use in rocky and master?14:34
odyssey4melogan-: is that fix in stable-2.4 ?14:34
noonedeadpunkand actually why stop/start of one rabbitmq instance inside cluster paralize work of the whole cluster14:34
logan-odyssey4me: hunting that down now14:35
odyssey4mevrobert: could you register a bug please, and state the performance impact - for Queens I think only the hardening role used that... but for rocky we have a bunch14:37
vrobertthank you very much guys!14:37
vrobertI think rocky is not affected because its using ansible v2.5 not v.2.4. I read that lot of mem problems fixed in 2.5...14:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update ansible to 2.4.6  https://review.openstack.org/61359914:38
vrobertbut I cant test it...14:38
odyssey4meyeah, but I do think include vs import needs attention - so the bug would still be appreciated14:39
odyssey4meusing include rather than import breaks tag usage for any tags in the subtasks, which isn't good14:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/queens: Update ansible to 2.4.6  https://review.openstack.org/61360014:40
logan-the comments in the bug lead me to believe some fixes were backported to 2.4, but there are no PRs linked to the issue so its hard to track down individual commits14:41
logan-the delta between 2.4.6 -> stable-2.4 does not appear to have anything performance related, so getting to 2.4.6 should catch any fixes that were bp14:42
vrobertGuys I tested with 2.4.6 but the results were the same...:(14:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: Update ansible to 2.3.4  https://review.openstack.org/61360314:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update ansible to 2.3.4  https://review.openstack.org/61360414:44
odyssey4melogan-: that's assuming there was a useful fix at all, so ideally we should fix our things to do better14:44
logan-yeah14:44
odyssey4mevrobert: we'll do the fix, but need a bug14:44
vrobertokay, sure14:45
vrobertI will fill a bug report on this.14:45
odyssey4methanks vrobert14:51
cloudnullmornings14:54
*** DanyC has joined #openstack-ansible14:55
*** DanyC_ has quit IRC14:59
ThiagoCMCMorning!15:00
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible master: Integrate Blazar with OpenStack Ansible  https://review.openstack.org/54995615:01
ThiagoCMCBTW cloudnull, I tried nspawn, setup-hosts.yml works great! All containers have connectivity and etc (macvlan is awesome). But, setup-infrastructure.yml failed (at utility), I tried to ignore and proceed but, only problems later on... I'm giving up on it for a while.   :-/15:02
cloudnullbuuu.15:03
cloudnullwhat were the issues with the utility container ?15:03
*** dave-mccowan has joined #openstack-ansible15:04
spotzmorning15:08
*** dave-mccowan has quit IRC15:10
vrobertodyssey4me: Bug report has been filled: https://bugs.launchpad.net/openstack-ansible/+bug/180016915:10
openstackLaunchpad bug 1800169 in openstack-ansible "ansible-hardening role runs out of memory when running for ~60 nodes" [Undecided,New]15:10
*** cshen has quit IRC15:12
odyssey4methank vrobert!15:14
cloudnullodyssey4me done.15:15
odyssey4mecloudnull: thanks - busy verifying adding lists vs union, and whether union takes multiple args15:15
cloudnullThiagoCMC - error: 503: Service Unavailable - seems like an issue with haproxy15:15
*** hamzaachi has quit IRC15:19
*** ThiagoCMC has quit IRC15:20
vrobertNo problem, thank you guys!15:22
odyssey4mecloudnull: list1 | union(list2) | union(list3) = (list1 + list2 + list3) | unique ... but the second is perhaps a little easier to read?15:22
*** gyee has joined #openstack-ansible15:23
cloudnull++ that works too15:23
*** vnogin has quit IRC15:25
*** vrobert has quit IRC15:27
*** ThiagoCMC has joined #openstack-ansible15:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add default distro packages for wheel builds  https://review.openstack.org/61358515:27
ThiagoCMCcloudnull, yeah, I thought about haproxy or it's backend (the repo_container) but, I'll just stick with LXC for now, it was fun to try nspawn! I learned a lot!  :-)15:28
cloudnullThiagoCMC ++15:28
ThiagoCMC=)15:28
cloudnullthanks foi giving it a go15:28
cloudnullits still new and there are lots of rough edges15:28
cloudnullso i greatly appreciate all the feedback and the willingness to give it a spin15:29
odyssey4mejrosser: I wonder if pkgconf could replace pkg-config? Busy reading https://github.com/pkgconf/pkgconf#comparison-of-pkgconf-and-pkg-config-dependency-resolvers15:30
jrosseri think setup.py for systemd-python specifically calls the pkgconf binary15:30
odyssey4mehahaha https://github.com/pkgconf/pkgconf#pkg-config-symlink15:30
jrosseroh thats sad15:31
*** fatdragon has joined #openstack-ansible15:31
ThiagoCMCcloudnull, I can give it more tries in a couple weeks! It would be great if you could access my lab to debug the playbook, I bet that we could fix those things quickly! I'm good in finding bugs!  Lol15:32
ThiagoCMCNo rush...   =)15:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia stable/rocky: Fix updating amphora images  https://review.openstack.org/61331515:34
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61293515:36
*** cshen has joined #openstack-ansible15:37
*** rgogunskiy has quit IRC15:45
openstackgerritMerged openstack/openstack-ansible-os_watcher stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61289915:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_cinder master: Add libpython2.7 as a required package.  https://review.openstack.org/61337515:52
openstackgerritMerged openstack/openstack-ansible-os_congress stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61292715:59
*** DoberMann has joined #openstack-ansible16:01
DoberMannhi, i'm tring to do some cleanup in containers, and i did use cleanup-cinder.yml. but each time i use anothe cinder playbook the (generated) inventory adds back "cinder_scheduler" containers, with new ID of course16:03
DoberMannwhat am i missing ?16:03
DoberMannand of course playbook fails to gather facts on non-existant containers16:03
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Add libpython2.7 as a required package  https://review.openstack.org/61342816:04
odyssey4meDoberMann: not sure where you got cleanup-cinder from? but the approach is to remove it from openstack_user_config, then delete the containers using lxc-containers-destroy.yml, then to use inventory-manage.py to remove it from the inventory16:05
odyssey4meman, we really need to add a doc entry for that in the operations guide - DoberMann could you register a bug for that?16:06
*** ThiagoCMC has quit IRC16:07
*** vnogin has joined #openstack-ansible16:08
DoberMannodyssey4me: https://github.com/openstack/openstack-ansible/blob/stable/queens/scripts/upgrade-utilities/playbooks/cleanup-cinder.yml and it seems it does what you say (except from the openstack_user_config part)16:08
odyssey4meDoberMann: yeah, that's specifically for an upgrade and is not for general purpose usage16:09
DoberMannthis is an upgrade16:09
odyssey4meThe reason it doesn't do the openstack_user_config bit is because you're not actually removing cinder from the env - just the containers which no longer need to exist because of the consolidation16:10
DoberMannbut i do have to "replay" some instal playbook16:10
odyssey4meok, I don't think that play is properly idempotent - I've not had much chance to go back and verify that... you may have to remove them by hand if it fails the first time16:11
odyssey4mein that case it's just do the things in those steps without adjusting openstack_user_config16:11
odyssey4meyeah, it'll fail on a second run because the container will likely be offline16:12
odyssey4meI guess it could be made idempotent by doing some sort of connectivity check, then skipping those tasks - or perhaps doing some other sort of failure tolerance.16:13
*** vnogin has quit IRC16:13
DoberMannyes, but this is not problematic (unless it fails in the middle as it happened to me : had to manually execute removing from inventory part)16:13
DoberMannodyssey4me: thx i've missed cinder_scheduler occurences in /etc/openstack_deploy/env.d/cinder.yml16:17
*** hamzaachi has joined #openstack-ansible16:20
*** shardy has quit IRC16:20
openstackgerritGerman Eichberger proposed openstack/openstack-ansible-os_octavia master: Break out config files for each octavia service  https://review.openstack.org/60322816:24
odyssey4meDoberMann: hmm, yes - if you have your own env.d files then they'll need adjusting16:29
DoberMannthey're not "my own" but must have come with openstack-ansible upgrade16:35
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61293016:35
DoberManni'm upgrading Liberty->Queens (and steps by Mitaka, Newton , Ocata and Pike for cinder/nova/haproxy when needed)16:36
odyssey4meDoberMann: oh really? Ah yes - if you're coming from anything before newton then the process copies the in-tree ones over16:37
DoberManncurrently all in Queens on ubuntu 14.04 and rebuilding containers/venv in 16.0416:37
odyssey4meDoberMann: if you compare them all and they're the same, you can just empty the /etc/openstack_deploy/env.d folder16:37
DoberMannso i need to reuse os-*install to deploy on brand new 16.04 containers16:38
*** nsmeds_ has joined #openstack-ansible16:40
odyssey4meDoberMann: to remind us that we should https://bugs.launchpad.net/openstack-ansible/+bug/180018516:41
openstackLaunchpad bug 1800185 in openstack-ansible "Upgrades from <Newton carry unnecessary user-space env.d files" [Undecided,New]16:41
odyssey4meDoberMann: these are the notes by others who did the trusty->xenial conversion: https://etherpad.openstack.org/p/osa-newton-xenial-upgrade16:42
odyssey4meDoberMann: wait, you're running queens on trusty? not sure how given that ISA doesn't support that16:43
*** nsmeds_ has quit IRC16:45
DoberMannall "old services on old containers" work in trusty, newly created containers are create in 16.04 ( unique cinder_api_ , ....)16:45
DoberMann*created16:45
DoberMannthxi'll follow this bug16:46
*** nsmeds_ has joined #openstack-ansible16:46
*** rgogunskiy has joined #openstack-ansible16:46
odyssey4meDoberMann: heh, sounds like you're in for an exciting weekend :)16:46
odyssey4meI know that mgariepy, antonym and cloudnull have been involved in marathon upgrade sessions.16:47
cloudnull^ always a good time :)16:47
DoberMannit's not that hard, just long ;)16:47
*** luksky has quit IRC16:48
*** DanyC_ has joined #openstack-ansible16:48
DoberMannthe longuest part was to deal with (both with systemd) upstart init scripts16:48
cloudnullyea its lots of watching bits fly by16:48
odyssey4meI know that folks have been using https://github.com/openstack/openstack-ansible-ops/tree/master/leap-upgrades for that sort of thing - or borrowing concepts from it16:48
DoberMannaka backport some newton files/templates/handlers16:48
odyssey4meit's a terrible thing, but for a one-off it works well enough16:48
*** rgogunskiy has quit IRC16:50
DoberManni'm currently working on dev/staging environments the real production migration will be later16:50
*** DanyC has quit IRC16:51
DoberMannupgrading one of the repo container in xenial firest, to build both 14.04 and 16.04 venvs16:51
DoberMann*first16:51
odyssey4meThe approach in the leaps is pretty much - turn almost the whole control plane off, delete almost all containers (neutron agents stay up for L3), then run db migrations through each release and run any OSA upgrade things related to vars and such through each release, then finally build new containers then delete the last old ones once the L3 routers have moved over16:52
DoberManni'll try to rebuild nova liberty on 16.04 also to first migrate the compute nodes (avoiding libvirt version issue which is coming later)16:52
*** DanyC_ has quit IRC16:53
DoberManni'm not that violent but it does minimize downtime my way16:53
*** DanyC has joined #openstack-ansible16:53
*** hamzaachi has quit IRC16:54
DoberMannall queens version running in all containers, and then rolling upgrade 14.04->16.04 for all hosts/containers by deleting them one at a time16:54
odyssey4meyep, that works best16:55
odyssey4methe purpose of the leaps was really to jump from juno/kilo/liberty to newton in one shot, so downtime was made part of the expectation16:55
DoberMannand also neutron migration to bare metal in advance16:55
odyssey4mecontrol-plane down time that is16:55
odyssey4meah yes, very good idea16:56
odyssey4mesounds like you're doing all the right things!16:56
DoberManni hope so =)16:56
DoberMann(and rabbitmq cluster migration in advance also to avoid breaking it by renaming hostnames)16:58
odyssey4mecores - two release blocker fixess for rocky's next release are up - please review asap: https://review.openstack.org/613560 & https://review.openstack.org/61332716:59
DoberManndid not manage to avoid downtime with galera cluster though16:59
odyssey4meDoberMann: well, we did just manage to find a bug in the pike galera builds which was causing havoc16:59
cloudnullDoberMann im not sure if 14>16 upgrades have the same issue but 16>18 upgrade has broken venvs on compute hosts. wrote a simple bash function to rewire the venv allowing the services to restart post upgrade. https://gist.github.com/cloudnull/cb87440c8221104ed2b857e67289905f17:00
odyssey4meyou should be able to jump from newton to pike at least, maybe even straight to queens17:00
*** DanyC_ has joined #openstack-ansible17:00
cloudnullmight be useful, if you have the same issue, and not already solved the problem elsewhere.17:00
DoberMannodyssey4me: in fact only cinder has to be fully upgraded step by step, i jump some with nova17:00
cloudnullodyssey4me newton > pike works, newton > queens did not (at least in test).17:01
DoberMannand all other component are upgraded directly17:01
odyssey4meDoberMann: also, for Keystone Ocata->Pike there's a need to flush the memcache cache or you'll get a bunch of errors, so https://review.openstack.org/#/q/Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840 will help there17:01
odyssey4mecloudnull: I was suggesting newton->queens for galera_server only17:03
*** DanyC_ has quit IRC17:03
*** DanyC has quit IRC17:03
openstackgerritMerged openstack/openstack-ansible-os_octavia stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61291117:03
odyssey4meto make that work though, I think we had to shut down the two secondary nodes first so that only one was running... otherwise the galera upgrade resulted in comms failures due to the mixed versions17:04
openstackgerritNiko Smeds proposed openstack/openstack-ansible-galera_server master: Fix Galera self-signed SSL functionality  https://review.openstack.org/61341217:05
DoberManncloudnull: thx looks a lot like my 14.04 venv not working on 16.04 but i found the workaround by building venv for both 14.04 ane 16.04 version (upgrading one of the repo)17:05
DoberMannhave a nice WE, bye17:07
DoberMannthx17:07
nsmeds_Question for y'all: submitted first PR, feedback was given, updated PR. Do I need to go back to feedback and click "Done" on all the comments or ?17:08
nsmeds_the comments no longer show up on the updated patch set17:08
openstackgerritMerged openstack/openstack-ansible-repo_build stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61293417:09
odyssey4mensmeds_: if you've done them, no worries - replying 'Done' is optional... if you took a different approach or did nothing, then better to reply on the comments related on the old patch set17:10
odyssey4meok folks, I'm out for the w/end - have a great one!17:11
nsmeds_ok, thanks =)17:11
odyssey4mecores - before I go, I suspect that apt failure might hit the other branches too, so these would be advisable to merge soon: https://review.openstack.org/#/q/I7a50a64a4759865bceccc6ae02132663c328172b17:13
*** mmercer has joined #openstack-ansible17:14
kaiokmoodyssey4me: just a question: regarding https://bugs.launchpad.net/openstack-ansible/+bug/180016917:21
openstackLaunchpad bug 1800169 in openstack-ansible "ansible-hardening role runs out of memory when running for ~60 nodes" [Undecided,New]17:21
kaiokmohow bad is to change from include to import?17:21
*** vakuznet has joined #openstack-ansible17:21
logan-kaiokmo: it will require some thinking if there is a when conditional on the include task. otherwise it should have little to no downside17:23
kaiokmologan-: ah, I see.17:25
*** electrofelix has quit IRC17:32
*** lbragstad is now known as elbragstad17:37
*** DanyC has joined #openstack-ansible17:49
*** hamzaachi has joined #openstack-ansible18:05
*** DanyC has quit IRC18:10
*** medberry is now known as med_18:31
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: SUSE: Add system-user-nobody for Leap 15  https://review.openstack.org/61354718:51
openstackgerritMerged openstack/ansible-role-python_venv_build stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61293719:06
*** nsmeds_ has quit IRC19:07
*** noonedeadpunk has quit IRC19:08
openstackgerritMerged openstack/openstack-ansible-os_rally stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61292819:17
*** dave-mccowan has joined #openstack-ansible19:18
openstackgerritMerged openstack/openstack-ansible-tests master: test-log-collect: Improve rsync performance for log collection  https://review.openstack.org/61357319:26
*** nsmeds_ has joined #openstack-ansible19:29
*** cshen has quit IRC19:35
openstackgerritMerged openstack/openstack-ansible-tests stable/rocky: Restart lxc-dnsmasq after clearing iptables rules  https://review.openstack.org/61327019:42
*** cshen has joined #openstack-ansible20:01
*** ansmith_ has quit IRC20:12
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/queens: Add memcache flushing handler on db migrations  https://review.openstack.org/61248220:15
vakuznetreview please https://review.openstack.org/#/c/612011/20:20
*** spatel has joined #openstack-ansible20:24
*** hamzaachi has quit IRC20:36
*** spatel has quit IRC20:39
*** openstackstatus has quit IRC20:42
*** openstack has joined #openstack-ansible20:46
*** ChanServ sets mode: +o openstack20:46
*** spatel has joined #openstack-ansible20:58
*** spatel has quit IRC21:02
*** jamesdenton has quit IRC21:11
openstackgerritMerged openstack/openstack-ansible-os_nova master: Cleanup nova distro package install lists  https://review.openstack.org/61206121:13
openstackgerritMerged openstack/openstack-ansible-os_monasca-agent stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61290721:17
*** nsmeds_ is now known as nsmeds21:19
*** nsmeds has quit IRC21:21
*** nsmeds has joined #openstack-ansible21:25
*** nsmeds has quit IRC21:25
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61293121:26
*** jbadiapa has quit IRC21:34
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Stop arbitrary shutting down services  https://review.openstack.org/60622121:41
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Convert role to use a common systemd service role  https://review.openstack.org/60622221:41
openstackgerritAntony Messerli proposed openstack/openstack-ansible master: Restart cinder-scheduler instead of reloading  https://review.openstack.org/61369421:44
jrossersomething is very broken here http://logs.openstack.org/60/613560/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/20c50d8/logs/ara-report/21:48
jrosserit does bootstrap-aio *twice*21:48
* jrosser finished for today - anyone with time to look at that ^ please do21:49
logan-thats weird21:50
logan-theres only 1 run of bootstrap-aio in the console log21:50
logan-and the job ran successfully to completion21:50
*** strattao has quit IRC21:51
*** cshen has quit IRC21:53
jrosserThere are signs of deploy=upgrade, that’s the only clue I saw21:57
logan-when you look at the zuul ara report it is even more wtf http://logs.openstack.org/60/613560/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/20c50d8/ara-report/21:57
logan-the console log does not match up with the zuul ara at all21:58
*** cshen has joined #openstack-ansible21:59
logan-looking at the zuul log, you are right, the failed bootstrap-aio definitely ran AFTER the job in the console log.. all of the roles were already cloned: http://logs.openstack.org/60/613560/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/20c50d8/ara-report/result/69dbd1c1-f0a4-4e90-9f21-141d03e2a509/22:02
*** vakuznet has quit IRC22:04
*** ansmith_ has joined #openstack-ansible22:16
*** nsmeds has joined #openstack-ansible22:24
*** cshen has quit IRC22:25
openstackgerritMerged openstack/openstack-ansible master: Aodh group vars has wrong admin and internal endpoint url  https://review.openstack.org/61352122:44
*** cshen has joined #openstack-ansible22:50
cloudnullevenings :)23:00
*** pcaruana has quit IRC23:10
*** gyee has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!