Tuesday, 2020-12-29

*** rfolco has quit IRC00:02
*** rfolco has joined #openstack-ansible00:03
*** spatel has joined #openstack-ansible00:22
*** spatel has quit IRC00:36
*** tosky has quit IRC00:39
*** macz_ has joined #openstack-ansible00:40
*** macz_ has quit IRC00:44
*** rfolco has quit IRC01:51
openstackgerritzhangboye proposed openstack/openstack-ansible-os_barbican master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/76861601:55
openstackgerritzhangboye proposed openstack/openstack-ansible-os_tempest master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/76861802:02
openstackgerritzhangboye proposed openstack/openstack-ansible-os_swift master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/76861902:04
openstackgerrityang yawei proposed openstack/openstack-ansible master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible/+/76862002:04
openstackgerritzhangboye proposed openstack/openstack-ansible-rsyslog_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_server/+/76862102:07
openstackgerrityang yawei proposed openstack/openstack-ansible-os_keystone master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/76862302:11
openstackgerritzhangboye proposed openstack/openstack-ansible-os_sahara master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/76862402:13
openstackgerrityang yawei proposed openstack/openstack-ansible-tests master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76862502:14
openstackgerritzhangboye proposed openstack/openstack-ansible-os_rally master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/76862602:16
openstackgerritXinxin Shen proposed openstack/openstack-ansible-os_nova master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/76862702:19
openstackgerritzhangboye proposed openstack/openstack-ansible-os_panko master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/76862802:20
openstackgerritzhangboye proposed openstack/openstack-ansible-os_masakari master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/76862902:22
openstackgerrityang yawei proposed openstack/openstack-ansible-os_glance master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/76863002:23
openstackgerritXinxin Shen proposed openstack/openstack-ansible-os_aodh master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/76863102:23
openstackgerritzhangboye proposed openstack/openstack-ansible-os_heat master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/76863202:24
openstackgerritzhangboye proposed openstack/openstack-ansible-os_ceilometer master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/76863302:26
openstackgerritXinxin Shen proposed openstack/openstack-ansible-apt_package_pinning master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/76863402:27
openstackgerritzhangboye proposed openstack/openstack-ansible-nspawn_hosts master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/76863502:33
openstackgerrityang yawei proposed openstack/openstack-ansible-os_cloudkitty master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/76863602:34
openstackgerritXinxin Shen proposed openstack/openstack-ansible-apt_package_pinning master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/76863402:36
openstackgerritXinxin Shen proposed openstack/openstack-ansible-os_aodh master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/76863102:37
openstackgerritzhangboye proposed openstack/openstack-ansible-nspawn_container_create master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-nspawn_container_create/+/76863702:37
openstackgerrityang yawei proposed openstack/openstack-ansible-os_cinder master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/76863802:37
openstackgerritXinxin Shen proposed openstack/openstack-ansible-os_nova master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/76862702:37
openstackgerritzhangboye proposed openstack/openstack-ansible-os_horizon master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/76863902:39
*** klamath_atx has joined #openstack-ansible02:39
openstackgerrityang yawei proposed openstack/openstack-ansible-openstack_hosts master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/76864002:40
*** macz_ has joined #openstack-ansible02:40
openstackgerritzhangboye proposed openstack/openstack-ansible-os_neutron master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/76864102:41
openstackgerritzhangboye proposed openstack/openstack-ansible-os_manila master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/76864202:43
openstackgerrityang yawei proposed openstack/openstack-ansible-os_designate master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/76864302:44
*** macz_ has quit IRC02:45
openstackgerritzhangboye proposed openstack/openstack-ansible-os_zun master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/76864402:45
openstackgerrityang yawei proposed openstack/openstack-ansible-openstack_openrc master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/76864502:46
openstackgerritzhangboye proposed openstack/openstack-ansible-os_octavia master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/76864602:49
openstackgerrityang yawei proposed openstack/openstack-ansible-os_blazar master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/76864702:49
openstackgerritXinxin Shen proposed openstack/openstack-ansible-apt_package_pinning stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/76864802:49
openstackgerritzhangboye proposed openstack/openstack-ansible-os_magnum master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/76864902:51
openstackgerrityang yawei proposed openstack/openstack-ansible-haproxy_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76865102:55
openstackgerrityang yawei proposed openstack/openstack-ansible-ceph_client master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/76865202:58
openstackgerrityang yawei proposed openstack/openstack-ansible-os_tacker master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/76865403:00
openstackgerrityang yawei proposed openstack/openstack-ansible-os_ironic master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/76865503:05
openstackgerritzhangboye proposed openstack/ansible-hardening master: Add Victoria release info  https://review.opendev.org/c/openstack/ansible-hardening/+/76865603:06
openstackgerrityang yawei proposed openstack/openstack-ansible-os_trove master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/76865703:07
openstackgerrityang yawei proposed openstack/openstack-ansible-os_placement master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/76865803:09
openstackgerrityang yawei proposed openstack/openstack-ansible-os_gnocchi master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/76865903:12
*** stduolc has quit IRC04:48
*** stduolc has joined #openstack-ansible04:48
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
openstackgerritzhangboye proposed openstack/openstack-ansible-os_mistral master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/76866305:49
openstackgerritzhangboye proposed openstack/openstack-ansible-openstack_hosts master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/76866405:54
openstackgerritzhangboye proposed openstack/openstack-ansible-lxc_hosts master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/76866505:56
openstackgerritzhangboye proposed openstack/openstack-ansible-haproxy_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76866606:03
openstackgerritzhangboye proposed openstack/openstack-ansible-plugins master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/76866706:06
openstackgerritzhangboye proposed openstack/openstack-ansible-openstack_openrc master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/76866906:11
openstackgerritzhangboye proposed openstack/openstack-ansible-lxc_container_create master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/76867006:12
openstackgerritzhangboye proposed openstack/openstack-ansible-rabbitmq_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/76867106:16
openstackgerritzhangboye proposed openstack/openstack-ansible-repo_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/76867406:27
openstackgerritzhangboye proposed openstack/openstack-ansible-memcached_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/76867506:29
openstackgerrityang yawei proposed openstack/openstack-ansible-os_keystone master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/76867606:32
openstackgerritzhangboye proposed openstack/openstack-ansible-ceph_client master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/76867706:36
openstackgerrityang yawei proposed openstack/openstack-ansible-os_glance master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/76867806:36
openstackgerritzhangboye proposed openstack/ansible-role-systemd_service master: Add Victoria release info  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/76867906:42
openstackgerritzhangboye proposed openstack/ansible-role-systemd_networkd master: Add Victoria release info  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/76868006:44
openstackgerrityang yawei proposed openstack/openstack-ansible-tests master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76868106:48
openstackgerritzhangboye proposed openstack/openstack-ansible-os_tacker master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/76868506:59
openstackgerritzhangboye proposed openstack/ansible-role-python_venv_build master: Add Victoria release info  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/76868907:03
*** stduolc has quit IRC07:24
*** stduolc has joined #openstack-ansible07:27
*** jawad_axd has quit IRC07:37
*** jawad_axd has joined #openstack-ansible08:38
*** shyamb has joined #openstack-ansible09:15
*** shyam89 has joined #openstack-ansible09:15
*** shyam89 has quit IRC09:16
*** shyamb has quit IRC09:16
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Fix documentation links  https://review.opendev.org/c/openstack/openstack-ansible/+/76871909:20
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Fix doc links  https://review.opendev.org/c/openstack/openstack-ansible/+/76872009:26
*** lixingxing has joined #openstack-ansible09:52
*** lixingxing has left #openstack-ansible09:52
*** jawad_axd has quit IRC10:48
*** jawad_axd has joined #openstack-ansible10:49
*** jawad_axd has quit IRC10:56
openstackgerritErik Berg proposed openstack/openstack-ansible-lxc_hosts master: Use proxy if any proxy_vars are set  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/76872310:59
*** cp- has quit IRC11:34
*** cp- has joined #openstack-ansible11:39
*** cp- has quit IRC11:55
*** cp- has joined #openstack-ansible11:55
*** rfolco has joined #openstack-ansible11:57
*** tosky has joined #openstack-ansible12:05
*** spatel has joined #openstack-ansible12:13
*** klamath_atx has quit IRC12:17
*** klamath_atx has joined #openstack-ansible12:18
*** spatel has quit IRC12:18
*** macz_ has joined #openstack-ansible12:49
*** macz_ has quit IRC12:54
admin022rc1 .. fails with  The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required.    on keystone -- create database for service13:13
admin0i thought this was already patched13:13
admin0https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/717381/1/vars/debian.yml13:15
*** ygk_12345 has joined #openstack-ansible13:15
admin0sorry .. its there13:15
admin0it was another package13:15
admin0i recall facing this same issue before and manually fixing it13:16
ygk_12345hi all13:16
ygk_12345i am facing this error in horizon 20.0.2 branch13:16
ygk_12345http://paste.openstack.org/show/801326/13:16
ygk_12345can someone look into it13:16
ygk_12345is it a bug ?13:17
ygk_12345in Train horizon ?13:17
*** stduolc has quit IRC13:30
*** stduolc has joined #openstack-ansible13:31
*** SecOpsNinja has joined #openstack-ansible13:40
noonedeadpunkdo you have multidomain enabled? I think we were probably fixing this issue13:45
ygk_12345noonedeadpunk yes it it enabled13:46
noonedeadpunkI think you might be missing that patch https://opendev.org/openstack/openstack-ansible-os_horizon/commit/e3326b77086b3865202cff9bce95377937aa6f0713:47
noonedeadpunkoh, maybe it's https://opendev.org/openstack/openstack-ansible-os_horizon/commit/e2ef8afb990696a5ff4efefc357b504b92719b4e13:49
noonedeadpunkand we have not backported it to train at all...13:49
ygk_12345noonedeadpunk do u think its multidomain issue ? it worked earlier. I am just reinstalling13:50
noonedeadpunkyou have exactly same error as was claimed to in e2ef8afb990696a5ff4efefc357b504b92719b4e commit13:51
noonedeadpunkbut it seems that the first one already includes this fix and I think it was not presetn in 20.0.213:52
ygk_12345noonedeadpunk so what part of code should I add to local_settinggs.py file ?13:52
noonedeadpunkyou should actually just cherry-pick this commit to your tree13:52
noonedeadpunkit anyway makes multidomain usaqble13:53
kleiniI have a new phenomenon with 21.2.0: If I add a new router with the web frontend including setting the external gateway, the router works. If I do the same with command line tools and the external gateway is there set in an additional request, the router namespace is missing the default route on the external network. Did somebody see a similar problem?14:02
*** spatel has joined #openstack-ansible14:02
*** ygk_12345 has quit IRC14:02
noonedeadpunkno, not really.. also route is created only on active router, while backup ones stays intact14:11
spatelnoonedeadpunk: hey14:11
noonedeadpunko/14:11
spatelHATop task was keep failing until specified delegate_to host so as per your comment look like remote_src not working14:12
spatelIt was failing on /opt/cache/files directory doesn't exist14:12
noonedeadpunkwell, I don't say that everything is ok with hatop (honestly have no idea), but offered patch does not do proper thing. There's literally no sense in having hatop on deploy host because it won't show anything, since there's no haproxy on deploy host?14:14
spatelnoonedeadpunk: agreed but hatop also support TCP remote socket also :)14:15
noonedeadpunkbut I had't time to check it more deeply14:15
noonedeadpunkdo we configure haproxy to listen on tcp socket ?:)14:15
spatelYou won't see this issue if deploying AIO (This issue is only for multi-node deployment)14:15
noonedeadpunkand which one haproxy you should connect to?:)14:15
spatelnoonedeadpunk: i don't think we do that at present (i am just saying it does support remote socket)14:16
spatelwe don't need to do that, lets just find out why its failing and not getting deploy on haproxy14:16
noonedeadpunkyeah, I got it. I just mean that we need better solution to properly deploy hatop on hosts where it's intended to be at the moment14:16
noonedeadpunkyeah, exactly what I was trying to say :)14:16
spatelwhy don't we directly untar on haproxy node instead of first deploy on deployment-host14:17
spatelwget | untar14:17
noonedeadpunkI think that was exactly idea of remote_src14:17
noonedeadpunkwget because we didn't want to download it multiple times, and for ppl with limited connectivity (ie proxy) in the middle14:18
spatelThis task creating /opt/cache/files on deployment-host https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/tasks/haproxy_install.yml#L2714:18
noonedeadpunkbut we probably can wget it multiple times. however I don't understand why remote_src does not work though...14:19
spatelremote_src: "{{ (haproxy_hatop_downloader == 'deployment-host') | ternary('no', 'yes') }}"14:19
spatelI have fresh deployment, i can destroy my haproxy and try to re-build it again to verify14:20
noonedeadpunkok, maybe we need to `Ensure haproxy_hatop_download_path exists` for both delegate_to and not?14:20
noonedeadpunkor, I'd say do with delegate_te only when `haproxy_hatop_downloader == 'deployment-host'`14:21
spatelthat is what i am trying to understand if we don't specify delegate_to then how it will know14:21
noonedeadpunkyep, sounds like it's our issue, that we don't create destination directory14:21
spatelYes we need to create dir on haproxy also14:22
spatelcurrently we are just doing on deployment-host and that is why its failing14:22
noonedeadpunkyeah14:22
kleiniI can fix these routers by mirgrating them to a different network node. the migration then creates the necessary default route. this is kind of ugly14:25
spatelnoonedeadpunk: are you going to patch it that missing part ?14:26
noonedeadpunkI thought you will adjust your patch, but I can do that as well ofc)14:27
spateli love to do that so let me try.. first14:28
*** ygk_12345 has joined #openstack-ansible14:37
ThiagoCMCHello! The Major Upgrade `./scripts/run-upgrade.sh` is failing (Ussuri -> Victoria), error: http://paste.openstack.org/ <- TASK [Clone git repos (parallel)] - Any clue?15:00
ThiagoCMCI tried from both `22.0.0.0rc1` and `stable/victoria`, same result.15:01
ThiagoCMCI'm on Ubuntu 20.04.115:02
ThiagoCMCI also tried `./scripts/bootstrap-ansible.sh`, same error.15:03
noonedeadpunkcan you paste error?:)15:03
ThiagoCMCIt's right there!  =P15:03
ThiagoCMCOh, damn15:03
ThiagoCMC1 sec15:03
ThiagoCMChttp://paste.openstack.org/show/801332/15:04
noonedeadpunkjrosser: ^15:05
ThiagoCMCSoon as the Ussuri AIO deployment finished without error, I followed: https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-upgrades.html and saw that error.15:06
noonedeadpunkyeah, I think I saw that one and have some guesses about it15:06
ThiagoCMCOk, cool!15:07
jrosserwell senlin does not have a stable/ussuri branch15:07
ThiagoCMCBTW: `git version 2.25.1`15:07
ThiagoCMCHmm15:07
jrosserassuming its a U->V upgrade15:07
ThiagoCMCYes, it is.15:07
ThiagoCMCFresh Ussuri AIO -> Victoria = fail.15:07
openstackgerritSatish Patel proposed openstack/openstack-ansible-haproxy_server master: Fix HATop for haproxy  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76861515:08
jrosserThiagoCMC: contributions welcome! this is after all 'release candidate' which is where all these things need fixing15:10
noonedeadpunkas quick fix I can suggest to failover to the old git method15:11
noonedeadpunkbut probably worth patching our new parallel clone module to not fail that way...15:11
jrosserit's more about the data passed to it i guess15:11
jrosserif it's being asked to check out a not existing branch15:12
jrosserbut fail-and-continue behaviour would be good for this15:12
noonedeadpunkBut it fails on literally every repo15:13
noonedeadpunk`["Failed to reset /etc/ansible/roles/os_masakari\nCmd('git') failed due to: exit code(129)\n  cmdline: git reset --force --hard 9cd72c1ad374bb6b6bc113d797d72c9aa44e1e22\n  stderr: 'error: unknown option `force'`15:14
noonedeadpunkos_masakari was present for U :)15:14
noonedeadpunkI think that ubuntu 20.04 has never git that does not have an argument we're trying to pass it15:15
ThiagoCMCjrosser, I would LOVE to be able to help! But I don't know if I have the required skill to fix it.  :-P15:15
noonedeadpunks/never/newer/15:15
jrosser--force is a fallback in that module i think15:15
jrosserif something fails it trys again with --force15:15
jrosserwhich is a bit misleading in the error15:16
noonedeadpunkwell on machine with 20.04 it says for me `error: unknown option `force'`15:16
noonedeadpunkhttp://paste.openstack.org/show/801333/15:17
spatelThiagoCMC: glad you decided to use AIO first to verify upgrade..15:17
ThiagoCMCspatel, riiiight?  LOL15:17
jrossernoonedeadpunk: it took so long to merge that patch the options could indeed be out of date now15:18
noonedeadpunkFor 18.04 I believe they are still valid...15:18
*** sshnaidm|rover is now known as sshnaidm|afk15:34
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Set force option to false by default  https://review.opendev.org/c/openstack/openstack-ansible/+/76872915:34
ThiagoCMCwow, that was fast! I'll test it!  =P15:40
* noonedeadpunk testing now15:42
noonedeadpunkjrosser: the main problem here that we don't test this code in CI :(15:42
noonedeadpunkbut symlink repos15:43
noonedeadpunkwhich is valid for ci, but uh....15:43
ThiagoCMCJust curious... The upgrade procedure isn't tested as a whole?15:43
noonedeadpunkit is tested15:45
noonedeadpunkexcept cloning roles, since we need to get them from CI instead of cloning on our own...15:45
noonedeadpunkeventually focal is just broken even without upgrade for V15:45
ThiagoCMCYou mean, CentOS?  lol15:46
ThiagoCMCkidding15:46
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Set force option to false by default  https://review.opendev.org/c/openstack/openstack-ansible/+/76872915:47
spatelnoonedeadpunk: don't say that focal is broken.. lol (i just moved my production to focal)15:48
*** noonedeadpunk has quit IRC15:48
ThiagoCMC:-D15:49
spatelQuestion for compute node do you keep /var/lib/nova on separate partition?15:49
ygk_12345hi all15:49
ThiagoCMCspatel, I don't15:50
spatelare you doing flat single root part?15:50
ygk_12345i am facing  an error for train horizon dashboard. when multidomain support is enabled it is throwing errors15:50
*** noonedeadpunk has joined #openstack-ansible15:50
ygk_12345http://paste.openstack.org/show/801326/15:50
ygk_12345i applied the patch as well. but it is not working15:50
ygk_12345https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/719934/2/templates/horizon_local_settings.py.j215:50
ThiagoCMCspatel, yes, always, just 1 single primary partition, everywhere. Sometimes, on top of a Linux mdadm (raid1) volume (also single primary part).15:51
noonedeadpunkThiagoCMC: check please updated one15:51
noonedeadpunkI missed closing bracet in the first version15:51
ThiagoCMCnoonedeadpunk, yes, I saw that =P15:52
ThiagoCMCfatal: [localhost]: FAILED! => {"msg": "template error while templating string: unexpected '}', expected ')'. String: {{ not (ansible_distribution | lower == 'ubuntu' and ansible_distribution_version is version('20.04', '>=') }}"}15:52
ThiagoCMCBut it's proceeding! With next task...15:52
ThiagoCMCI'm not sure if I have to control+c it15:52
noonedeadpunkyeah, you can15:53
ThiagoCMCdone lol15:53
ThiagoCMCLooks like that it worked!15:56
ThiagoCMCThe "TASK [Clone git repos (parallel)]" just returned one line: "changed: [localhost]" - nothing else.15:57
ThiagoCMCI'm not seeing it cloning Victoria repos.15:57
ThiagoCMCMaybe it cloned before with "TASK [Clone git repos (with git)]" when I hit control+c. I'll try again from a fresh Ussuri AIO.15:59
noonedeadpunknah, it just does not have output like previously was16:00
ThiagoCMCHmm ok16:01
noonedeadpunkshould we have a meeting today? It's just holidays time so was thinking if we should ruin mood with looking through bugs :p16:01
ygk_12345I have filed a bug for now16:03
ygk_12345https://bugs.launchpad.net/horizon/+bug/190958716:03
openstackLaunchpad bug 1909587 in OpenStack Dashboard (Horizon) "Multidomain issue" [Undecided,New]16:03
ThiagoCMCI'm kinda good in finding bugs!  lol16:03
noonedeadpunktbh I'd vote to skip meeting today tbh and do after NY16:05
spatel+116:08
spatelThiagoCMC: i am planning to create separate /var/lib/nova to protect system when run out of disk or something16:09
spatelonly worried about logs fill up disk16:09
spateldoes nova use /var/lib/nova to report available disk space?16:09
ThiagoCMCSure, sounds cool... I'm not sure if Nova uses that subdir/mount point to calculate the free space.16:12
spatelwhat nova use to calculate disk space if we have multiple partitions ?16:13
ThiagoCMCtrue16:13
ThiagoCMC=P16:13
*** chandankumar is now known as raukadah16:28
*** noonedeadpunk has quit IRC16:40
*** noonedeadpunk_ has joined #openstack-ansible16:48
*** macz_ has joined #openstack-ansible16:51
*** macz_ has quit IRC16:56
spatelThiagoCMC: does upgrade works for you?17:27
spatelnoonedeadpunk_: what it will take to merge HATop commit? - https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/768615/2/tasks/haproxy_install.yml17:33
spatelits kind of blocker at present17:33
spatelspecially in multi-node scenario17:34
ThiagoCMCIt's working on the AIO, still on it17:38
ThiagoCMCMy haproxy in prod is separated17:39
spatelif haproxy is separate then you need that patch to move forward17:39
ThiagoCMCThanks!17:40
ThiagoCMCSo far, Victoria still needs 2 patches...17:40
ThiagoCMC1 for when upgrading from Ussuri, another for when haproxy is isolated.17:41
jrosserspatel: i am not sure the patch is right17:42
spateljrosser: hmm could you please explain what is not right?17:44
jrosserI left a comment17:44
jrosserwhat happens when the targethost is the downloader and run_once=true?17:45
noonedeadpunk_replied...17:46
noonedeadpunk_well...17:46
noonedeadpunk_in any scenario we download to the single host17:46
noonedeadpunk_but without run_once we do this 3 times?17:46
ThiagoCMCspatel, my haproxy is from Ubuntu repos. I'm not sure if I'll hit that code...17:47
*** ygk_12345 has quit IRC17:48
spatelThiagoCMC: its not about haproxy installer issue, its HATop utility issue.17:48
ThiagoCMCOh, ok =P17:50
spatelnoonedeadpunk_: why don't we remove run_once=true17:51
spatelits pretty small binary not going to take long to download and even if it fail that is ok17:51
noonedeadpunk_because then we will run get_url 3 times - we are delegating same task for each host we're running against to the same `haproxy_hatop_downloader`17:52
noonedeadpunk_why don't we just drop all this complexity with `haproxy_hatop_downloader` and dowload directly to haproxy hosts - well, I'm kind of about to agree to this scenario17:53
noonedeadpunk_jrosser: what do you think?17:53
spatelagreed (even we can just drop whole hatop and move this to openstack-ansible-ops), I don't know how many people using hatop17:54
noonedeadpunk_considering it was always there - I think max is to install it conditionally17:54
jrosserit is too complex currently17:56
spatelnoonedeadpunk_: lets make it simple like look into inventory and if you find haproxy component then deploy directly17:56
jrosserjust having a url to override the download location should cover pretty much everything17:56
noonedeadpunk_and we do have this already17:57
jrosserI think we use that, I’ll have to check17:58
noonedeadpunk_haproxy_hatop_download_url ?17:58
noonedeadpunk_we would need to write reno in case we drop `haproxy_hatop_downloader` variable17:59
noonedeadpunk_including upgrade one17:59
jrosserwe set haproxy_hatop_downloader: target-host and haproxy_hatop_download_url: "http://thing-accessible-in-dmz/hatop-0.7.7.tar.gz"18:02
noonedeadpunk_then in your case you delegate download task to each target host several times or what? uhhhhhhh18:03
noonedeadpunk_I'd say let's really drop haproxy_hatop_downloader then18:04
noonedeadpunk_I think we don't use this anywhere else when we need to download smth anyway18:04
noonedeadpunk_*this approach18:04
jrosserno we don't18:04
spatelI doubt anyone here knows we do have that utility also :)18:05
spatelif people need that they can install manually because its very easy to do18:05
noonedeadpunk_spatel: do you want to change your patch to reflect that?18:07
spatelnoonedeadpunk_: tell me what exactly we are doing to do ?18:08
*** jawad_axd has joined #openstack-ansible18:10
noonedeadpunk_dropping haproxy_hatop_downloader variable and all related to it stuff18:10
noonedeadpunk_we can make hatop installation conditional as well18:10
ThiagoCMCThe `hatop` is not even working here in my Ussuri.18:11
spatelso remove haproxy_hatop_downloader and let it download directly to target host right?18:11
noonedeadpunk_yep18:11
* noonedeadpunk_ never tried to use it18:12
ThiagoCMCError: http://paste.openstack.org/show/801336/18:12
spatelhatop is extra headache to maintain (In Ussuri it doesn't support python3 version)18:12
ThiagoCMCOuch18:12
ThiagoCMCFeels like dead code to me18:13
noonedeadpunk_we probably should backport new version to U18:13
spatelthat is totally possible to do18:13
spatellet me try to re-adjust my patch18:13
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/ussuri: Python3 supported version of hatop  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76873918:14
*** macz_ has joined #openstack-ansible18:18
spatelwhat to do with this part  - https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/tasks/haproxy_install.yml#L4818:23
spateljust hardcode "Yes" ?18:23
noonedeadpunk_yep18:24
spatelgood18:24
*** jawad_axd has quit IRC18:24
noonedeadpunk_well, other option it to always download to the deploy host actually:)18:24
noonedeadpunk_but whatever :)18:25
noonedeadpunk_I'm ok with any of that18:25
noonedeadpunk_no idea which is better18:25
spatelwhat about this - http://paste.openstack.org/show/801337/18:25
spatelwe should remove run_once: true18:25
ThiagoCMCOr make an PPA  =P18:26
ThiagoCMCBTW, hatop was removed from Debian testing/unstable.18:26
jrosserthat command/creates is probably not good for upgrades either18:26
jrossernot sure why not to just use copy module18:27
spatellet me try to use copy module18:35
spateljrosser: noonedeadpunk_ does it looks good ? - http://paste.openstack.org/show/801338/18:55
spatelchanged command to copy (so it will help for upgrade)18:55
noonedeadpunk_let's probably also add boolean variable haproxy_hatop_install : true and place this under block with condition?19:00
noonedeadpunk_run_once is also not needed19:00
spatelwhere do you want me to place haproxy_hatop_install: true? (under copy?)19:01
spatelyou want to give option to end-users to decide (yes/no)19:03
spatelnoonedeadpunk_: let me try my way and let me know if its good or bad19:10
noonedeadpunk_I think it should be placed to defaults and all these tasks should be under block with condition19:10
noonedeadpunk_yeah, so we have easy workaround in case we got issues with deployment again19:11
spatelI like it..19:11
spatellet me put them inside block:19:12
*** jamesdenton has quit IRC19:16
*** jamesdenton has joined #openstack-ansible19:25
openstackgerritSatish Patel proposed openstack/openstack-ansible-haproxy_server master: Fix HATop for haproxy  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76861519:30
spatelnoonedeadpunk_: jrosser check that out and let me know if need any cosmetic changes19:30
noonedeadpunk_1. release note, 2. can't we just unarchive to somewhere like /usr/local/bin so we don't need to copy?19:38
noonedeadpunk_spatel: ^19:38
spatelnoonedeadpunk_: hmmm19:42
spateldoes unarchive support extracting single file? or we can use single command line tar xvzf foo.tar.gz bin/hatop something19:43
openstackgerritSatish Patel proposed openstack/openstack-ansible-haproxy_server master: Fix HATop for haproxy  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76861519:49
spatelnoonedeadpunk_: check release notes19:49
admin0i have a zombie cinder-api-container-9090427e    listed19:54
admin0how do I delete it ?19:54
*** macz_ has quit IRC19:55
spatelanything hold in lsof ?20:00
admin0hmm.. let me reboot that system20:04
noonedeadpunk_spatel: and why you replaced install with copy? I guess install should place man as well?20:12
noonedeadpunk_but whatever...20:12
spatelnoonedeadpunk_: if in future we want to upgrade then install won't work20:12
spateljrosser: suggested that option20:12
noonedeadpunk_fair20:12
admin0didn't help20:13
spatelanyway its single python script20:13
admin0that entry is still there20:13
spateli don't think anyone worried about man :)20:13
spatelI think we can go ahead with this method because its much simpler than earlier.20:14
spateladmin0: are you still not able to remove container?20:14
admin0nope20:14
admin0its an entry of a container that no longer exists20:15
spatelstop/destroy doesn't work?20:15
admin0its not the container20:16
admin0the container is not there20:16
spateli thought you said "cinder-api-container-9090427e" container in zombie20:17
admin0https://snipboard.io/rbtvjO.jpg  -- this one spatel20:17
admin0that container does not even exist20:18
admin0  Admin >> System >> System Information >> Block Storage Service20:19
admin0is where that output is from20:19
spatelmay be somewhere in config file or empty folder20:19
spatellxc-ls -f not showing right?20:20
admin0its not in the lxc-ls -f20:21
admin0its not in the inventory as well20:21
spatelvery odd20:21
spatelfolder /var/lib/lxc/<dir> ?20:22
admin0hmm.. its not in /var/lib/lxc, but its inside /openstack/20:23
spatelget rid of it20:24
admin0and reboot ?20:25
admin0but it says down, so i am looking for a way to clear it from the db20:25
*** jawad_axd has joined #openstack-ansible20:36
admin0figured it out .. cinder-manage service remove cinder-scheduler cloud-c1-1030989-cinder-api-container-9090427e20:38
*** jawad_axd has quit IRC20:42
ThiagoCMCQuick question! Is it going to be easier to use OSA Victoria with Ceph and Erasure Coded pools?20:46
spateli didn't try anything related ceph on victoria but it should be fine.20:48
spateladmin0: nice20:48
*** SecOpsNinja has left #openstack-ansible20:48
admin0osa 22.1 beta  fails at os_keystone : Create database for service  -- with msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required20:49
admin0i am on ubuntu 2020:49
spatelyesterday i did install 22.0.0.rc1 on 20.04.1 and it works fine20:49
admin0mine is failing at that step .20:50
spateldo you have python3 installed?20:50
admin0is that a manual step ?20:50
spatelno20:50
admin0i did a min install and then just run the playbooks20:50
admin0min install only with ssh and nothing else20:50
spatelworth looking full error logs20:51
admin0well, but that galera is inside a container20:51
spatelhttps://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html20:51
spatelstep 3. saying make sure python3 is installed20:52
spatelit should be there in minimum install20:52
admin0spatel, https://gist.githubusercontent.com/a1git/0ae319976c9edf6adc2fccc0fcba7216/raw/9a33deaafe6078e7533f8431496e8d8fb3d55d16/gistfile1.txt20:53
spatel{{ install_method }}_install.yml' due to an undefined variable20:54
admin0spatel, those are already in all the hosts20:54
spatelI do have install_method: source20:54
spatelin user_variable.yml file20:54
admin0i mean those packkages are there20:54
admin0i don't have that in my user_variables20:55
admin0a new requirement ?20:55
spatelI do have install_method: source  (not sure why)20:55
admin0i do not have it20:55
spatelskipping vars_file 'defaults/{{ install_method }}_install.yml' due to an undefined variable20:55
spatellook like you need that.. why don't you try20:55
admin0i see it, but i always ignored :)20:56
spatellooking playbook to understand logic20:56
spateldefault is install_method: source but some reason i put it there.. not sure why21:00
spateli don't put option without reason :)21:01
spatelanyway run playbook with that option and if it works then we can ask jrosser or noonedeadpunk_21:01
admin0same error:  "msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required."21:02
admin0python3-pymysql is there inside the galera container21:04
spatelin logs its saying - <c1> Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 202021:06
spatelmay be same DNS issue which i had21:07
spatelcan you ping c1 from your deployment host ?21:07
spatelits trying to use utility container to create database on mysql21:08
spatelI remember when i had same issue i did following to fix it but in victoria that option is default21:10
-spatel- #openstack_service_setup_host: "{{ groups['utility_all'][0] }}"21:10
-spatel- #openstack_service_setup_host_python_interpreter: "/openstack/venvs/utility-{{ openstack_release }}/bin/python"21:10
admin0i can .. its 172.29.236.11 and added in /etc/hosts as well21:12
spatelYour utility container trying to use default python to create database ( It should use /openstack/venvs/utility-{{ openstack_release }}/bin/python)21:13
spatelcan you ssh to db box from utility container?21:13
admin0that does not work ever21:13
admin0coz utility does not have the id_rsa keys21:14
spatelwhy not?21:14
admin0only deploy hash21:14
admin0only deploy has21:14
admin0you can mysql ENTER from deploy container to get to mysql directly21:14
admin0one thougt is that it only has python3-mysql and maybe i need to add python-mysql in the debian.vars of the galera playbook and check21:15
spateltry that two line in user_variable.yml and see if that fix issue21:15
spatelI had same issue in past21:16
admin0which 2 lines ?21:17
admin0oh21:17
admin0those21:17
admin0one moment21:17
spatelyes openstack_service_setup_host... blah21:17
admin0didn't21:18
admin0so that python3-mysql -- is that needed on galera server or in keystone container ?21:19
spatelit should be need on galera21:20
admin0i am going to destroy the lab and switch back to 21.2.0 :)21:20
spatelwhy i didn't see that error and you are having lots of issue21:21
admin0will test 22 again in a future date21:21
spatelwhat version are you installing?21:21
admin022.0.0.0rc121:21
spatelIt should work.. because it works for me without mucking21:21
spatelverify your containers are good and function because i had small issue, some of my container didn't get ip etc.. and solution was reboot machine21:23
*** sri_ has quit IRC21:28
*** sri_ has joined #openstack-ansible21:29
*** rfolco has quit IRC21:43
*** jawad_axd has joined #openstack-ansible21:58
*** jawad_axd has quit IRC21:59
jrosserpython3-mysql shouldnt be needed on the galera container21:59
jrosserservice & db setup defaults to the utility container for V22:00
*** stduolc has quit IRC22:00
*** stduolc has joined #openstack-ansible22:00
spateljrosser: could you please sign off on this patch because its block deployment - https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76861522:07
spatelnoonedeadpunk_: ^^22:07
jrosseri don't want to rush22:08
jrossermake a fork locally22:08
jrosserremember that you are deploying a stable branch22:09
jrosserso even if we merge now / backport / merge to victoria22:09
jrosserthe role sha will not be updated until the next V rc release22:09
spatelYes that is true, just to verify or review sign off otherwise we can find other way to deploy.. just code validation22:10
*** rfolco has joined #openstack-ansible22:11
jrosserthere isnt a release note?22:12
*** rfolco has quit IRC22:12
spatelI never did that so not sure how to do that part22:13
*** rfolco has joined #openstack-ansible22:13
spateli can find some example22:13
jrosserthere is a tool called 'reno' which generates a template for you22:16
jrosserhttps://docs.openstack.org/reno/latest/user/usage.html22:16
*** rfolco has quit IRC22:18
openstackgerritMerged openstack/openstack-ansible-haproxy_server master: Add Victoria release info  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76866622:21
spatelthank you jrosser (reading)22:22
jrosserbasically 'reno new some-useful-few-words'22:23
jrosserthen edit what it makes to end up with something like what is in here https://github.com/openstack/openstack-ansible-haproxy_server/tree/master/releasenotes/notes22:23
spateljrosser: i will do "reno new some-useful-few-words" inside haproxy_server repo right?22:25
jrosseryep22:25
spatelgot it so it will drop that works inside releasenotes/notes/  directory22:26
jrosserobviously change some-useful-few-words to be meaningful :)22:26
jrosserthats right22:26
spateltotally does it support multi-line ?22:26
spatelor pass some text file or yaml file?22:26
spateli can see some example - reno new slug-goes-here --from-template my-file.yaml22:27
spatelI will figure out :)22:27
spateljrosser: after adding reno, i should do new git commit -m "release notes" right?22:29
spatelso it will create new commit request22:29
spatelI gotta go, i will catch you tomorrow22:30
spatelThank for the help22:30
*** spatel has quit IRC22:30
*** jawad_axd has joined #openstack-ansible22:58
*** jawad_axd has quit IRC22:59
noonedeadpunk_jrosser: btw bump tool is failing for v because of https://review.opendev.org/c/openstack/openstack-ansible/+/76861123:08
noonedeadpunk_but thinking about it now I think I need to double check if it the only issues23:09
noonedeadpunk_seems it is, yes23:11
*** klamath_atx has quit IRC23:13
*** klamath_atx has joined #openstack-ansible23:13
*** jawad_axd has joined #openstack-ansible23:21
*** jawad_axd has quit IRC23:24
*** klamath_atx has quit IRC23:31
*** rfolco has joined #openstack-ansible23:44

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!