| *** ansmith has joined #openstack-ansible | 00:04 | |
| *** ianychoi has quit IRC | 00:07 | |
| *** ianychoi has joined #openstack-ansible | 00:08 | |
| *** gyee has quit IRC | 00:13 | |
| *** tosky has quit IRC | 00:28 | |
| *** aedc_ has joined #openstack-ansible | 00:50 | |
| *** ondrejburian has quit IRC | 00:50 | |
| *** ondrejburian has joined #openstack-ansible | 00:51 | |
| *** aedc has quit IRC | 00:52 | |
| *** ianychoi has quit IRC | 01:30 | |
| *** ianychoi has joined #openstack-ansible | 01:32 | |
| *** ianychoi has quit IRC | 02:10 | |
| *** ianychoi has joined #openstack-ansible | 02:18 | |
| *** weifan has joined #openstack-ansible | 03:33 | |
| *** weifan has quit IRC | 03:35 | |
| *** ianychoi has quit IRC | 03:35 | |
| *** ianychoi has joined #openstack-ansible | 03:37 | |
| *** prometheanfire has quit IRC | 03:47 | |
| *** prometheanfire has joined #openstack-ansible | 04:10 | |
| *** shyamb has joined #openstack-ansible | 04:45 | |
| *** ianychoi has quit IRC | 05:00 | |
| *** ianychoi has joined #openstack-ansible | 05:03 | |
| *** udesale has joined #openstack-ansible | 05:07 | |
| *** raukadah is now known as chandankumar | 05:18 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-ansible | 05:35 | |
| *** aedc_ has quit IRC | 06:01 | |
| *** ianychoi has quit IRC | 06:01 | |
| *** aedc has joined #openstack-ansible | 06:02 | |
| *** ianychoi has joined #openstack-ansible | 06:03 | |
| *** ianychoi has quit IRC | 06:11 | |
| *** ianychoi has joined #openstack-ansible | 06:13 | |
| *** shyamb has quit IRC | 06:20 | |
| *** ianychoi has quit IRC | 06:29 | |
| *** ianychoi has joined #openstack-ansible | 06:31 | |
| *** ianychoi has quit IRC | 06:39 | |
| *** ianychoi has joined #openstack-ansible | 06:41 | |
| *** ianychoi has quit IRC | 06:58 | |
| *** shyamb has joined #openstack-ansible | 06:59 | |
| *** ianychoi has joined #openstack-ansible | 07:05 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:16 | |
| *** jbadiapa has joined #openstack-ansible | 07:22 | |
| *** miloa has joined #openstack-ansible | 07:23 | |
| miloa | morning | 07:23 |
|---|---|---|
| *** shyam89 has joined #openstack-ansible | 07:28 | |
| miloa | exit | 07:30 |
| miloa | oups sorry :) | 07:30 |
| *** miloa has quit IRC | 07:30 | |
| *** shyamb has quit IRC | 07:32 | |
| *** shyam89 has quit IRC | 07:46 | |
| *** aedc has quit IRC | 07:47 | |
| *** miloa has joined #openstack-ansible | 07:53 | |
| *** pcaruana has joined #openstack-ansible | 08:16 | |
| *** tosky has joined #openstack-ansible | 08:28 | |
| *** gkadam has joined #openstack-ansible | 08:34 | |
| *** gkadam has quit IRC | 08:34 | |
| *** cshen has joined #openstack-ansible | 08:43 | |
| *** gouthamr has quit IRC | 08:49 | |
| *** gouthamr has joined #openstack-ansible | 08:50 | |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** shyamb has joined #openstack-ansible | 09:15 | |
| *** udesale_ has joined #openstack-ansible | 09:27 | |
| *** udesale has quit IRC | 09:30 | |
| *** cshen has quit IRC | 09:38 | |
| *** pcaruana has quit IRC | 09:46 | |
| noonedeadpunk | o/ | 09:46 |
| jrosser | morning | 09:50 |
| * jrosser looks at ujson 2.0.1 release as next broken thing.... | 09:51 | |
| *** cshen has joined #openstack-ansible | 10:05 | |
| *** Blinkiz has quit IRC | 10:15 | |
| *** Blinkiz has joined #openstack-ansible | 10:16 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat elasticsearch monitoring on Elastic hosts https://review.opendev.org/712314 | 10:20 |
| -openstackstatus- NOTICE: The mail server for lists.openstack.org is currently not handling emails. The infra team will investigate and fix during US morning. | 10:25 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat elasticsearch monitoring on Logstash hosts https://review.opendev.org/712318 | 10:26 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat logstash monitoring on Logstash hosts https://review.opendev.org/712318 | 10:27 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group https://review.opendev.org/707849 | 10:31 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat elasticsearch monitoring on Elastic hosts https://review.opendev.org/712314 | 10:31 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat logstash monitoring on Logstash hosts https://review.opendev.org/712318 | 10:31 |
| *** fghaas has joined #openstack-ansible | 10:39 | |
| *** rh-jelabarre has joined #openstack-ansible | 10:50 | |
| *** shyam89 has joined #openstack-ansible | 11:03 | |
| *** shyamb has quit IRC | 11:07 | |
| *** cshen has quit IRC | 11:13 | |
| *** udesale_ has quit IRC | 11:17 | |
| *** rh-jelabarre has quit IRC | 11:18 | |
| *** ianychoi has quit IRC | 11:24 | |
| noonedeadpunk | are broken things going to end?:) | 11:32 |
| *** aedc has joined #openstack-ansible | 11:34 | |
| *** cshen has joined #openstack-ansible | 11:35 | |
| jrosser | noonedeadpunk: here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8b2/711821/12/check/openstack-ansible-deploy-aio_lxc-debian-buster/8b228a8/logs/ara-report/result/50dd8bc6-c270-4e0e-8836-2dcf824507d4/ | 11:35 |
| *** xakaitetoia has joined #openstack-ansible | 11:38 | |
| *** nicolasbock has joined #openstack-ansible | 11:38 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove python2 or python3 problem pip packages https://review.opendev.org/712069 | 11:44 |
| *** shyamb has joined #openstack-ansible | 11:48 | |
| *** ansmith has quit IRC | 11:48 | |
| *** shyam89 has quit IRC | 11:50 | |
| *** shyamb has quit IRC | 11:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add Journalbeat support for OS LXC containers https://review.opendev.org/711934 | 11:54 |
| *** rpittau is now known as rpittau|bbl | 11:58 | |
| *** mkuf_ has joined #openstack-ansible | 11:59 | |
| *** aedc has quit IRC | 11:59 | |
| *** mkuf has quit IRC | 12:02 | |
| *** udesale has joined #openstack-ansible | 12:02 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:10 | |
| *** pcaruana has joined #openstack-ansible | 12:15 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Correctly enable mysql filebeat module https://review.opendev.org/712094 | 12:21 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat elasticsearch monitoring on Elastic hosts https://review.opendev.org/712314 | 12:21 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat logstash monitoring on Logstash hosts https://review.opendev.org/712318 | 12:21 |
| *** rh-jelabarre has joined #openstack-ansible | 12:28 | |
| *** mkuf has joined #openstack-ansible | 12:29 | |
| *** mkuf_ has quit IRC | 12:32 | |
| *** ansmith has joined #openstack-ansible | 12:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Install python3 tools and libraries instead of python2 https://review.opendev.org/710659 | 12:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-apt_package_pinning master: Updated from OpenStack Ansible Tests https://review.opendev.org/712358 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ceph_client master: Updated from OpenStack Ansible Tests https://review.opendev.org/712359 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_client master: Updated from OpenStack Ansible Tests https://review.opendev.org/712360 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712361 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-haproxy_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712362 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests https://review.opendev.org/711691 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/709471 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-memcached_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712363 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests https://review.opendev.org/712364 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/712365 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/712366 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_openrc master: Updated from OpenStack Ansible Tests https://review.opendev.org/712367 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ops master: Updated from OpenStack Ansible Tests https://review.opendev.org/712368 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.opendev.org/709479 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.opendev.org/712369 | 12:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests https://review.opendev.org/712370 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/709482 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.opendev.org/712371 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.opendev.org/712372 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.opendev.org/712373 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.opendev.org/712374 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.opendev.org/711704 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi master: Updated from OpenStack Ansible Tests https://review.opendev.org/709488 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.opendev.org/712375 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_horizon master: Updated from OpenStack Ansible Tests https://review.opendev.org/712376 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.opendev.org/712377 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.opendev.org/712378 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests https://review.opendev.org/711709 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.opendev.org/712379 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.opendev.org/712380 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.opendev.org/712381 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_murano master: Updated from OpenStack Ansible Tests https://review.opendev.org/712382 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.opendev.org/712383 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.opendev.org/712384 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.opendev.org/712385 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_panko master: Updated from OpenStack Ansible Tests https://review.opendev.org/712386 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_placement master: Updated from OpenStack Ansible Tests https://review.opendev.org/712387 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_rally master: Updated from OpenStack Ansible Tests https://review.opendev.org/712388 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.opendev.org/712389 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.opendev.org/712390 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.opendev.org/712391 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tempest master: Updated from OpenStack Ansible Tests https://review.opendev.org/712392 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.opendev.org/712393 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.opendev.org/709512 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-pip_install master: Updated from OpenStack Ansible Tests https://review.opendev.org/709513 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-plugins master: Updated from OpenStack Ansible Tests https://review.opendev.org/712394 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rabbitmq_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712395 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_build master: Updated from OpenStack Ansible Tests https://review.opendev.org/709516 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712396 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_client master: Updated from OpenStack Ansible Tests https://review.opendev.org/711726 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/709519 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-config_template master: Updated from OpenStack Ansible Tests https://review.opendev.org/709520 | 12:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-hardening master: Updated from OpenStack Ansible Tests https://review.opendev.org/671840 | 12:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests https://review.opendev.org/712397 | 12:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-role-systemd_mount master: Updated from OpenStack Ansible Tests https://review.opendev.org/712398 | 12:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-role-systemd_networkd master: Updated from OpenStack Ansible Tests https://review.opendev.org/712399 | 12:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-role-systemd_service master: Updated from OpenStack Ansible Tests https://review.opendev.org/712400 | 12:48 |
| *** aedc has joined #openstack-ansible | 12:52 | |
| djhankb | noonedeadpunk: Regarding https://review.opendev.org/#/c/705125/ - I was having issues with the swift venv build on Debian 9, it was failing with "Cannot find libsystemd or libsystemd-journal" and jrosser suggested that I add 'libsystemd-dev' to the os_swift/vars/debian.yml file, which fixed the issue, so I submitted that change. I apologize for | 13:12 |
| djhankb | using the channel here, as I'm not familiar enough with Gerrit to reply to your question on there. | 13:12 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server stable/train: lsyncd: use relative paths in exclude statement https://review.opendev.org/712093 | 13:13 |
| *** aedc has quit IRC | 13:14 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Updated from OpenStack Ansible Tests https://review.opendev.org/712368 | 13:18 |
| *** irclogbot_2 has quit IRC | 13:22 | |
| *** irclogbot_2 has joined #openstack-ansible | 13:22 | |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712362 | 13:25 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/712365 | 13:28 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests https://review.opendev.org/712364 | 13:30 |
| jrosser | noonedeadpunk: any ideas here? https://zuul.opendev.org/t/openstack/build/8b228a8b387d48f2bad28ba927d0d848/log/logs/openstack/aio1_utility_container-dcf6c07b/python_venv_build.log.txt#47339 | 13:32 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning master: Updated from OpenStack Ansible Tests https://review.opendev.org/712358 | 13:33 |
| * noonedeadpunk looking | 13:37 | |
| noonedeadpunk | djhankb: yeah, that's totally ok to use IRC:) | 13:38 |
| noonedeadpunk | djhankb: just would be great having issue in commit msg though. Can you kindly add `systemd-devel` package for redhat and suse then? | 13:40 |
| *** DanyC has joined #openstack-ansible | 13:40 | |
| noonedeadpunk | as if it's required for debian, it's pretty likely required for other distros as well | 13:40 |
| noonedeadpunk | jrosser: feels like it's compiling py2 stuff for py3.... | 13:41 |
| *** dave-mccowan has joined #openstack-ansible | 13:41 | |
| noonedeadpunk | like this is not py3 valid thing https://github.com/testing-cabal/testtools/blob/master/testtools/_compat2x.py#L16 | 13:43 |
| *** rh-jelabarre has quit IRC | 13:43 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:44 | |
| jrosser | hmm https://github.com/testing-cabal/testtools/blob/f865cfefae73e540aedb16ebb19f813cc648f2b6/NEWS#L539 | 13:44 |
| *** dave-mccowan has quit IRC | 13:46 | |
| noonedeadpunk | hm, what #941958 is.... | 13:48 |
| *** joshualyle has joined #openstack-ansible | 13:48 | |
| noonedeadpunk | ok, so https://github.com/testing-cabal/testtools/commit/56c49a5fd420143627e50e2d0ec86e1f37075875 | 13:49 |
| *** zigo has quit IRC | 13:49 | |
| *** joshualyle has quit IRC | 13:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_placement stable/train: Fix placement DB migration https://review.opendev.org/710786 | 13:55 |
| *** dave-mccowan has joined #openstack-ansible | 13:57 | |
| djhankb | noonedeadpunk: I can do that, whats the best process to amend my change, I don't necessarily want to create a new change, or do I? | 13:59 |
| noonedeadpunk | no, if you have your patch in local tree, just to changes, `git add .; git commit --amend; git review` | 13:59 |
| jrosser | if you've not got your patch handy you can do git review -d <change-id> and it will grab it and make you a local branch | 14:00 |
| noonedeadpunk | also during commit ammend you can change commit messagfe ;) | 14:00 |
| noonedeadpunk | jrosser: maybe it's some accidental failure? as ubuntu passes and there were no problems yestarday with debian as well? | 14:02 |
| jrosser | i'm getting closer...... http://paste.openstack.org/show/790542/ | 14:02 |
| jrosser | well, actually thats maybe something different | 14:03 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Use blockinfile for hosts file generation https://review.opendev.org/709153 | 14:04 |
| noonedeadpunk | you think we're missing g++? | 14:05 |
| djhankb | Ok thanks for the info, I'll get those added. | 14:05 |
| jrosser | noonedeadpunk: perhaps on buster we explicity need the g++ package | 14:05 |
| jrosser | i was just looking here https://github.com/openstack/ansible-role-python_venv_build/blob/master/vars/main.yml#L16-L35 | 14:06 |
| noonedeadpunk | like weird if that dependecy appeared just tonight... | 14:07 |
| jrosser | ujson got bumped in u-c very recently | 14:08 |
| noonedeadpunk | but yeah, I can imagine that | 14:08 |
| noonedeadpunk | yeah... | 14:08 |
| jrosser | https://github.com/openstack/requirements/commit/abe9c1b3ade21144e73e113ab466b67ed418c710#diff-0bdd949ed8a7fdd4f95240bd951779c8R394 | 14:09 |
| jrosser | and i think it fails on a dep of ujson | 14:09 |
| noonedeadpunk | and rally is the only who needs ujson by far?:) | 14:09 |
| noonedeadpunk | oh, indeed, it's bumped to new major release... | 14:09 |
| jrosser | bizzare, right?! | 14:10 |
| noonedeadpunk | yeah, agree, let's just install g++, but maybe only for rally role? | 14:10 |
| noonedeadpunk | btw, interestingly, that rally has it's own u-c https://opendev.org/openstack/rally-openstack/src/branch/master/upper-constraints.txt | 14:12 |
| noonedeadpunk | maybe we should use them instead? | 14:12 |
| openstackgerrit | Henry Bonath proposed openstack/openstack-ansible-os_swift master: add libsystemd-dev to swift_devel_distro_packages https://review.opendev.org/705125 | 14:13 |
| djhankb | Done | 14:13 |
| * noonedeadpunk wondering if rally will work with new ujson | 14:15 | |
| jrosser | noonedeadpunk: i think i'm missing something - woudlnt we expect to see these things https://github.com/openstack/ansible-role-python_venv_build/blob/master/vars/main.yml#L16-L35 installed in the repo server container? | 14:17 |
| jrosser | heres the repo server https://zuul.opendev.org/t/openstack/build/8b228a8b387d48f2bad28ba927d0d848/log/logs/openstack/aio1_repo_container-3c086e83/apt/history.log.txt | 14:18 |
| noonedeadpunk | actually I see that ujson is only used by monasca only... | 14:18 |
| noonedeadpunk | jrosser: so we would have gcc instaled, but probably not g++? | 14:18 |
| jrosser | heres the utility container https://zuul.opendev.org/t/openstack/build/8b228a8b387d48f2bad28ba927d0d848/log/logs/openstack/aio1_utility_container-dcf6c07b/apt/history.log.txt | 14:18 |
| jrosser | why do we see the toolchains installed in utility but not repo? | 14:19 |
| jrosser | i though that should all happen on the repo container https://github.com/openstack/ansible-role-python_venv_build/blob/90fe5472765931acd6764dfe73b121507b01f81a/tasks/python_venv_wheel_build.yml#L24 | 14:20 |
| noonedeadpunk | not really | 14:23 |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server stable/stein: lsyncd: use relative paths in exclude statement https://review.opendev.org/712456 | 14:25 |
| noonedeadpunk | oh, yes, you're right... | 14:25 |
| jrosser | here https://github.com/openstack/ansible-role-python_venv_build/blob/8a64d095833eff5a7db3c43665a52b82b1d6818e/vars/main.yml | 14:25 |
| noonedeadpunk | just missed indent | 14:25 |
| noonedeadpunk | hm... may it be because of block inside block? | 14:25 |
| *** cshen has quit IRC | 14:26 | |
| noonedeadpunk | but all previous tasks are not delegated as well | 14:27 |
| noonedeadpunk | jrosser: it's delegated actually https://zuul.opendev.org/t/openstack/build/8b228a8b387d48f2bad28ba927d0d848/log/job-output.txt#21746 | 14:28 |
| noonedeadpunk | seems ara don't have delegation info..... | 14:28 |
| noonedeadpunk | jrosser: so we probably need to replace gcc with g++ here https://github.com/openstack/ansible-role-python_venv_build/blob/8a64d095833eff5a7db3c43665a52b82b1d6818e/vars/main.yml#L19 | 14:30 |
| jrosser | i think we need both | 14:30 |
| jrosser | C and C++ compiler | 14:30 |
| jrosser | like for the other distros | 14:30 |
| jrosser | let me make a patch and we'll see | 14:31 |
| noonedeadpunk | gcc is strict requirement for g++ | 14:31 |
| noonedeadpunk | https://packages.debian.org/buster/g++ | 14:31 |
| noonedeadpunk | but not visa versa | 14:31 |
| jrosser | i wonder if gcc is in the container image which is why we don't see it in the apt history | 14:31 |
| noonedeadpunk | s/visa/vice/ | 14:31 |
| jrosser | sadly we don't grab a list of the apt state form the containers | 14:31 |
| noonedeadpunk | hm, that might be actually.... | 14:32 |
| jrosser | oh hold on, meybe we do | 14:32 |
| *** chandankumar has left #openstack-ansible | 14:35 | |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Ensure g++ is present on debian/ubuntu for wheel builds https://review.opendev.org/712459 | 14:36 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use zuul cached repos for openstack services https://review.opendev.org/711821 | 14:37 |
| *** sshnaidm|afk is now known as sshnaidm | 14:37 | |
| *** cshen has joined #openstack-ansible | 14:49 | |
| openstackgerrit | Merged openstack/openstack-ansible-plugins master: Updated from OpenStack Ansible Tests https://review.opendev.org/712394 | 14:51 |
| *** ahosam has joined #openstack-ansible | 14:57 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.opendev.org/712372 | 15:09 |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server stable/rocky: Exclude repo_build_global_links_path from sync https://review.opendev.org/705390 | 15:33 |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server stable/rocky: lsyncd: use relative paths in exclude statement https://review.opendev.org/712482 | 15:33 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: fix spelling of python package https://review.opendev.org/710781 | 15:36 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Move all tox environments to python3 https://review.opendev.org/710253 | 15:37 |
| *** gyee has joined #openstack-ansible | 15:47 | |
| *** zigo has joined #openstack-ansible | 15:51 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group https://review.opendev.org/707849 | 15:54 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Correctly enable mysql filebeat module https://review.opendev.org/712094 | 15:54 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat elasticsearch monitoring on Elastic hosts https://review.opendev.org/712314 | 15:54 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Enable Filebeat logstash monitoring on Logstash hosts https://review.opendev.org/712318 | 15:54 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: pin ceph packages to release o=Ubuntu on rocky/bionic https://review.opendev.org/708420 | 15:56 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712361 | 16:01 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712363 | 16:04 |
| *** joshualyle has joined #openstack-ansible | 16:16 | |
| *** joshualyle has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc master: Updated from OpenStack Ansible Tests https://review.opendev.org/712367 | 16:25 |
| *** cshen has quit IRC | 16:31 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.opendev.org/712378 | 16:37 |
| *** udesale has quit IRC | 16:41 | |
| *** rpittau|bbl is now known as rpittau | 16:43 | |
| *** spatel has joined #openstack-ansible | 16:50 | |
| spatel | Folks my rabbitMQ cluster is totally dead.. not services start stuck ether for long time | 16:50 |
| spatel | Do you think i should clean up /var/lib/rabbitmq/mnesia/ dir and start? | 16:51 |
| spatel | is that safe? | 16:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Updated from OpenStack Ansible Tests https://review.opendev.org/712392 | 16:57 |
| cjloader | so jrosser noonedeadpunk I don't think i'll have cycles to work on cloudkitty at this time | 16:58 |
| noonedeadpunk | rodger that | 16:58 |
| spatel | can someone just help me out with that? | 16:59 |
| spatel | is it safe to remove /var/lib/rabbitmq/mnesia/ and start cluster again? | 16:59 |
| noonedeadpunk | not sure.... have you tried running rabbitmq-setup -e rabbitmq_upgrade=true? | 17:00 |
| spatel | not yet... | 17:01 |
| noonedeadpunk | I always start with that :p | 17:01 |
| spatel | Thinking to start manually first before any playbook | 17:01 |
| spatel | All document saying remove /var/lib/rabbitmq/mnesia/ and start services | 17:02 |
| spatel | I think i have more compute nodes in cluster and that is why rabbitMQ not keeping up | 17:03 |
| spatel | https://access.redhat.com/solutions/1596043 | 17:03 |
| *** rh-jelabarre has quit IRC | 17:05 | |
| spatel | noonedeadpunk: rabbitmq-setup -e rabbitmq_upgrade=true? what is rabbitmq_upgrade=true going to do ? | 17:06 |
| spatel | is this going to upgrade thing? | 17:07 |
| noonedeadpunk | actually this will re-create cluster, queues, etc | 17:07 |
| *** rh-jelabarre has joined #openstack-ansible | 17:08 | |
| noonedeadpunk | upgrade will be performed in case you have another version set in vars (or do some upgrade) | 17:08 |
| spatel | Does it ok to re-create cluster in production ? | 17:09 |
| spatel | noonedeadpunk: ^^ | 17:09 |
| noonedeadpunk | like I've runned with that flag in prod in case of rabbit issues dozen of times | 17:11 |
| spatel | can you give me full command with playbook name ? | 17:11 |
| spatel | i think i need to run that because everything is broken at this point | 17:11 |
| noonedeadpunk | openstack-ansible /opt/openstack-ansible/playbooks/rabbitmq-install.yml -e rabbitmq_upgrade=true | 17:13 |
| spatel | running it now | 17:13 |
| spatel | do i need to save any data or anything ? | 17:13 |
| *** nicolasbock has quit IRC | 17:14 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client master: Updated from OpenStack Ansible Tests https://review.opendev.org/712360 | 17:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_placement master: Updated from OpenStack Ansible Tests https://review.opendev.org/712387 | 17:16 |
| spatel | noonedeadpunk: anyway running that playbook | 17:17 |
| *** miloa has quit IRC | 17:17 | |
| *** nicolasbock has joined #openstack-ansible | 17:20 | |
| spatel | noonedeadpunk: damn it.. it stuck here last 5 min -> TASK [rabbitmq_server : Lock package versions] ****** | 17:24 |
| noonedeadpunk | hm.... | 17:24 |
| spatel | do you think because of rabbitmq_upgrade=ture? | 17:24 |
| spatel | do you think i should try with rabbitmq_upgrade=false? | 17:25 |
| noonedeadpunk | what branch are you on? | 17:25 |
| spatel | Queens | 17:25 |
| noonedeadpunk | oh... | 17:25 |
| spatel | 16.x something | 17:26 |
| spatel | any problem? | 17:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.opendev.org/712369 | 17:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.opendev.org/712373 | 17:27 |
| spatel | I am worried what if it won't fix it :( | 17:27 |
| spatel | it already removed rabbitMQ services | 17:27 |
| noonedeadpunk | never tried it running on queens - only starting from rocky | 17:28 |
| spatel | :( | 17:28 |
| spatel | damn it.. | 17:29 |
| -spatel- https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: [Errno 12] Timeout on https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received') | 17:31 | |
| -spatel- Trying other mirror. | 17:31 | |
| -spatel- https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: [Errno 12] Timeout on https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: (28, 'Operation timed out after 30002 milliseconds with 0 out of 0 bytes received') | 17:31 | |
| -spatel- Trying other mirror. | 17:31 | |
| -spatel- https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: [Errno 12] Timeout on https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received') | 17:31 | |
| -spatel- Trying other mirror. | 17:31 | |
| -spatel- https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: [Errno 12] Timeout on https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received') | 17:31 | |
| -spatel- Trying other mirror. | 17:31 | |
| spatel | look like bunch of repos are broken | 17:31 |
| spatel | I thought rabbitMQ is local packages.. | 17:31 |
| noonedeadpunk | works for me though | 17:31 |
| jrosser | i can wget those url from here | 17:31 |
| noonedeadpunk | maybe you have some network issues? | 17:32 |
| spatel | check yourself... | 17:32 |
| spatel | does it work for you? | 17:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.opendev.org/712381 | 17:32 |
| spatel | Network is fine.. | 17:33 |
| spatel | I can ping google from DNS name | 17:33 |
| spatel | let me curl it | 17:33 |
| spatel | wget is working | 17:33 |
| spatel | In RabbitMQ.repo i have "proxy = http://172.28.0.9:3142" | 17:34 |
| spatel | do you think something is wrong there? | 17:34 |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-ansible | 17:35 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.opendev.org/711704 | 17:36 |
| spatel | noonedeadpunk: why we have proxy in rabbitMQ repo? | 17:37 |
| noonedeadpunk | not sure it's intentional | 17:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests https://review.opendev.org/712370 | 17:38 |
| spatel | I have "proxy = http://172.28.0.9:3142" setting in every repo | 17:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Updated from OpenStack Ansible Tests https://review.opendev.org/712388 | 17:39 |
| spatel | why does OSA using it ? | 17:39 |
| spatel | should i remove that and try ? | 17:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.opendev.org/712375 | 17:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.opendev.org/712380 | 17:44 |
| jrosser | spatel: hold on | 17:45 |
| jrosser | that is the repo server which maintans a package cache - depending how old code you are running | 17:46 |
| spatel | jrosser: so you think that is not my issue right? | 17:48 |
| jrosser | well it is probably exactly your issue, that you can't get those packages via the cache on the repo server | 17:49 |
| jrosser | but you can curl direct | 17:49 |
| spatel | jrosser: also i found "rdo-release.repo" file pointing to -> rdo-release.repo:baseurl = http://mirror.centos.org/centos/7/cloud/$basearch/openstack-pike/ | 17:49 |
| spatel | openstack-pike but i am running queens | 17:49 |
| spatel | jrosser: how do i fix my repo server? | 17:50 |
| jrosser | you will have to check but i beleive patches got backported to fix that | 17:50 |
| *** DanyC_ has joined #openstack-ansible | 17:50 | |
| cyberpear | They do a bad job of keeping the rdo-release up top date | 17:50 |
| spatel | should i change pike to queens ? | 17:51 |
| *** DanyC_ has quit IRC | 17:52 | |
| *** DanyC_ has joined #openstack-ansible | 17:52 | |
| spatel | jrosser: what are the option i have ? how do i fix my repo cache? | 17:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.opendev.org/712371 | 17:53 |
| spatel | do i need to clean cache? | 17:53 |
| *** DanyC has quit IRC | 17:54 | |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: Updated from OpenStack Ansible Tests https://review.opendev.org/712398 | 17:54 |
| spatel | I am running tcpdump on repo server and not getting any traffic from rabbitMQ container | 17:55 |
| openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.opendev.org/712393 | 17:55 |
| *** DanyC_ has quit IRC | 17:56 | |
| jrosser | spatel: i am just leaving the office now | 17:58 |
| spatel | :( | 17:58 |
| spatel | does rabbmitMQ create issue if its down for long time? | 17:58 |
| jrosser | but check you can ping, check you can curl / wget the repo server on the proxy port | 17:58 |
| spatel | from repo server i can access everything... | 17:58 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.opendev.org/712385 | 17:58 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.opendev.org/712391 | 17:58 |
| spatel | even i can install packages from repo server | 17:58 |
| jrosser | look at the history for the repo server and upgrade playbooks, we did cleanup for exactly this at some point but i can't dig that our for you right now | 17:59 |
| jrosser | ^ cleanup for removing the repo server proxy config from the hosts at upgrade | 17:59 |
| spatel | I am not seeing any traffic coming on repo server from rabbotMQ container (i am trying to install yum install foo for testing) | 17:59 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.opendev.org/712389 | 17:59 |
| spatel | jrosser: tell me last thing... does rabbitMQ outage cause any issue? | 18:00 |
| spatel | in existing cluster? | 18:00 |
| spatel | I can understand you can't create VM etc.. but nothing else can go wrong right? | 18:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.opendev.org/712384 | 18:00 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests https://review.opendev.org/712397 | 18:00 |
| spatel | This is massive production cluster and i will be done if this issue won't get fix | 18:00 |
| jrosser | if you need an emergency fix then remove the yum proxy config by hand | 18:01 |
| jrosser | then the repo server is not involved | 18:01 |
| spatel | let me try that | 18:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.opendev.org/712390 | 18:03 |
| *** sshnaidm is now known as sshnaidm|afk | 18:04 | |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712395 | 18:06 |
| spatel | jrosser: i found something | 18:08 |
| spatel | apt-cacher-ng.service was in zombie stats | 18:08 |
| spatel | my monitoring was showing its GREEN but it was not working | 18:09 |
| spatel | I have kill and restarted and now its working.. i can reach to all repo | 18:09 |
| spatel | re-running rabbitmq playbook | 18:09 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: Updated from OpenStack Ansible Tests https://review.opendev.org/712396 | 18:11 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Updated from OpenStack Ansible Tests https://review.opendev.org/712400 | 18:11 |
| jrosser | cores: need this to unblock buster jobs https://review.opendev.org/#/c/712459/ | 18:13 |
| spatel | Join rabbitmq cluster is failed | 18:15 |
| spatel | "stderr": "Error: mnesia_not_running", "stderr_lines": ["Error: mnesia_not_running"] | 18:16 |
| spatel | do i need to delete mnesia directory ? | 18:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_murano master: Updated from OpenStack Ansible Tests https://review.opendev.org/712382 | 18:18 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Update deprecated xpack monitoring settings https://review.opendev.org/712528 | 18:22 |
| spatel | After removing mnesia directory now i can join cluster successful | 18:28 |
| spatel | noonedeadpunk: do i need to restart all other services after rebuild new cluster? | 18:30 |
| noonedeadpunk | only stuck one | 18:31 |
| noonedeadpunk | but most of them should join | 18:31 |
| spatel | noonedeadpunk: i am not able to see any queue like /nova /neutron etc.. | 18:31 |
| spatel | it should create queue itself right? | 18:32 |
| spatel | before i had all queues | 18:32 |
| spatel | noonedeadpunk: who will create queues ? | 18:33 |
| spatel | rabbitmqctl list_queues is empty output | 18:33 |
| spatel | rabbitmqctl list_queues -p nova also empty | 18:33 |
| spatel | I check on GUI and no queue there | 18:34 |
| spatel | AMQPLAIN login refused: user 'neutron' - invalid credentials | 18:34 |
| *** macz_ has joined #openstack-ansible | 18:34 | |
| spatel | noonedeadpunk: do i need to run all other playbooks to create queue ? | 18:35 |
| spatel | Anyone can please reply :( | 18:36 |
| spatel | jrosser: Do you know how do i get all queue back in cluster ? | 18:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.opendev.org/712377 | 18:40 |
| spatel | jrosser: all queues are missing in rabbitMQ | 18:41 |
| jrosser | spatel: I am sat in a train with only phone internet :) | 18:41 |
| jrosser | iirc the osa roles each set up the rabbit vhost and users they need | 18:42 |
| spatel | jrosser: sorry, i am totally stuck. Just tell me do i need to run all playbook like nova, neutron in order to create rabbitMQ queue ? | 18:42 |
| jrosser | once those are there the sevices should create the queues | 18:43 |
| spatel | jrosser: i just re-build cluster but it has zero queue (you are saying it should get create automatically?) | 18:44 |
| jrosser | no | 18:44 |
| jrosser | each openstack service ansible role will make what it needs | 18:44 |
| spatel | jrosser: so i need to run each playbook right? | 18:45 |
| jrosser | so the quickest thing is proabably to rerun your playbooks just for the infra nodes | 18:45 |
| spatel | nova, neutron, heat etc.. | 18:45 |
| jrosser | or even just one of them | 18:45 |
| spatel | let me try to run heat | 18:45 |
| jrosser | then you visit all the services and the rabbit tasks should do the setup | 18:46 |
| jrosser | yes try one first | 18:46 |
| jrosser | but quite likely you don’t need to target all the computes | 18:46 |
| spatel | running -> openstack-ansible os-heat-install.yml | 18:46 |
| jrosser | you could check to see if there are tags which just do the rabbit setup | 18:47 |
| spatel | digging for that tag.. | 18:47 |
| spatel | wish this could be part of rabbiMQ document (like how to recover from issue like this ) | 18:48 |
| jrosser | would none of your rabbit nodes restart? | 18:48 |
| spatel | jrosser: my rabbitmq cluster is up and running (just missing queues) | 18:48 |
| jrosser | yes, I mean before you delete all the data | 18:49 |
| spatel | Yes my rabbitmq cluster was in zombie state (couldn't do anything) so i try to restart one node and it failed to start, then noonedeadpunk suggested to rebuild it . | 18:50 |
| spatel | jrosser: it just create /heat queue | 18:51 |
| spatel | i think i need to run all my playbooks | 18:51 |
| jrosser | awesome | 18:51 |
| spatel | can i just run them with --limit <rabbitmq_container> ? | 18:51 |
| jrosser | no, because heat play won’t target rabbit so it’s won’t do anything | 18:52 |
| jrosser | if there is a tag use that | 18:52 |
| spatel | looking for tag to limit that | 18:52 |
| jrosser | or —limit to “all the things on infra1” < you’ll need to find he right group for that | 18:53 |
| jrosser | noonedeadpunk: passing upgrades! https://review.opendev.org/#/c/712459/ | 18:56 |
| spatel | jrosser: i didn't find any tags which create rabbitMQ eueu | 18:58 |
| djhankb | spatel: I found a script I used awhile back when I screwed up my rabbitmq setup to re-create all of the queues and permissions: https://gitlab.com/snippets/1952141 | 19:02 |
| spatel | djhankb: where i should run this and what about password? | 19:03 |
| djhankb | if you look at the script, you can set the password for each service which you would get from user-secrets.yml | 19:03 |
| djhankb | then just run it inside your rabbitmq containers | 19:03 |
| djhankb | I would very carefully run it one line at a time to be sure that it's going to do what you want it to do | 19:04 |
| spatel | djhankb: This is cool... | 19:04 |
| spatel | let me try one line first and see | 19:04 |
| spatel | djhankb: do i need to run on each rabbitMQ ? | 19:05 |
| djhankb | spatel: I honestly don't remember if I had to do it on all or just one since they are clustered. | 19:05 |
| spatel | let me try one single and see it get replicate | 19:06 |
| djhankb | I just saw your comment that rabbitmq was working, but blank - and remembered I had made this | 19:06 |
| spatel | djhankb: i have thi in user_secret file "glance_rabbitmq_password: 2aaccdb7746966b4e5db5d2110fd225" | 19:07 |
| spatel | so this should be my password right? | 19:07 |
| djhankb | yes, thats also what would be in the glance config file as well | 19:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.opendev.org/712379 | 19:24 |
| spatel | djhankb: you saved me today with that small script | 19:25 |
| spatel | all you need to do is run on single node and it will auto replicate to other nodes | 19:26 |
| djhankb | Nice!! I'm super glad that helped | 19:26 |
| spatel | You should commit this in OSA (life-save) directory | 19:26 |
| djhankb | yeah, probably not a bad idea. It would be nice if it somehow read the variables from the user_variables.yml as that would be helpful. Not sure where something like this would go. | 19:27 |
| spatel | I would just put small document around that script and publish... | 19:31 |
| *** rpittau is now known as rpittau|afk | 19:40 | |
| spatel | Thank you everyone!!!!! it was fun day :( | 19:48 |
| spatel | specially jrosser & noonedeadpunk | 19:49 |
| spatel | soon i will move rabbitMQ from infra node to dedicated nodes | 19:49 |
| jrosser | spatel: great you are up and running again! I’m sure we could make a “rescue” tool of some sort | 19:51 |
| jrosser | openstack-ansible-ops repo is a good place for that sort of thing | 19:51 |
| *** fghaas has quit IRC | 19:51 | |
| spatel | jrosser: or just better documentation :) it took my breath away | 19:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: add libsystemd-dev to swift_devel_distro_packages https://review.opendev.org/705125 | 19:52 |
| spatel | I will push that script to openstack-ansible-ops for sure | 19:52 |
| djhankb | jrosser: I was just looking at that repo myself - I can go ahead and get it added there | 19:53 |
| spatel | djhankb: can you also please put small document in that script so people like me don't ask question | 19:54 |
| djhankb | yes, that was the plan - clean it up a little bit, add a README.md with some notes and then folks can use and improve upon it. | 19:54 |
| djhankb | oh sweet - that was my first ever OSA contribution that was just merged :-) | 19:56 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Ensure g++ is present on debian/ubuntu for wheel builds https://review.opendev.org/712459 | 19:56 |
| spatel | djhankb: glad you pop in just in time | 20:01 |
| djhankb | you bet, glad I was able to help. | 20:01 |
| jrosser | djhankb: do you need your swift patch on a stable release? | 20:03 |
| djhankb | jrosser: not at this time, I assume it'll bubble out when 20.0.2 is tagged? | 20:04 |
| jrosser | if it just merged to master you’ll need to cherry pick it to stable/train to get it into the next 20.x release | 20:06 |
| jrosser | you can do that in the gerrit web interface | 20:06 |
| djhankb | ah ok, I see that button there. I can give that a try. | 20:08 |
| spatel | djhankb: can you also add this command in your script to verify queue is there or not (rabbitmqctl list_queues -p /nova) | 20:09 |
| djhankb | spatel: sure. | 20:10 |
| djhankb | should we do something like: 'rabbitmqctl list_queues -p /nova | grep compute' and then check the result of "$?" before running the rabbitmq add_user/add_vhost ? That way it won't do anything if the user/vhost already exists | 20:14 |
| spatel | jrosser: My notes of today rabbitMQ activity -> http://paste.openstack.org/show/790563/ | 20:15 |
| spatel | djhankb: rabbitmqctl list_queues -p /nova will print big list of connected nodes.. ( this is just to verify your password etc.. is working) | 20:16 |
| djhankb | ah ok | 20:16 |
| djhankb | so those will not show anything if no hosts are connected? | 20:17 |
| spatel | djhankb: how about this command "rabbitmqctl list_users" | 20:21 |
| -spatel- # rabbitmqctl list_users | 20:21 | |
| -spatel- Listing users | 20:21 | |
| -spatel- heat[] | 20:21 | |
| -spatel- neutron[] | 20:21 | |
| -spatel- glance[] | 20:21 | |
| -spatel- monitoring[monitoring] | 20:21 | |
| -spatel- nova[] | 20:21 | |
| djhankb | yeah, was yours blank earlier? | 20:21 |
| spatel | It will be good verification instead of list_queue -p /nova (This command will take some time) | 20:21 |
| spatel | yes earlier it was blank | 20:21 |
| djhankb | Right, IIRC you said you have 100's of hosts - in my little test cluster I have 3 :-) | 20:22 |
| *** ansmith has quit IRC | 20:22 | |
| spatel | I have 338 compute nodes in cluster | 20:22 |
| spatel | list_queue -p /nova took time to print data | 20:22 |
| djhankb | nice. What are the specs of your controller(s) ? If I might ask | 20:23 |
| spatel | rabbitmqctl list_users this will be lightweight command to verify | 20:23 |
| spatel | Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz / 64GB / 500GB SSD (samsung) | 20:23 |
| djhankb | Ok right on - how many of those do you have? | 20:24 |
| spatel | 3 infra / 338 computes | 20:24 |
| djhankb | Ok awesome, that's not bad at all | 20:25 |
| spatel | I am not running ceilometer or any metering services. | 20:26 |
| spatel | I have 3 big cluster like that | 20:27 |
| spatel | each cluster has 300+ compute nodes | 20:27 |
| djhankb | oh cool. Yeah I'm just getting started- I'm about a year in to learning about Openstack. We run hosted VMs for our customers but we manage everything and it's all on VMWare or Hyper-V. I'm wanting to bring Openstack into the Mix to allow customers to self-manage where they would want to do that sort of thing. | 20:31 |
| djhankb | So I'm of course very interested as to what people's production setups look like. | 20:32 |
| spatel | i had same question before i build cluster... | 20:32 |
| spatel | My issue is, i had zero issue since i build so its now hard to know how to fix thing when it breaks... loosing muscle memory | 20:33 |
| spatel | I have to keep re-building cluster in lab to keep my memory up to date | 20:34 |
| djhankb | Yeah, I can totally understand being in that position. I built my first cluster about 8 months ago, then my second in January and it took quite a lot to get my head wrapped back around it. | 20:34 |
| djhankb | I just built another one, and am building upon what I've learned. But I can understand that once everything is stable just forgetting all of the little quirks | 20:35 |
| spatel | djhankb: ++ | 20:39 |
| spatel | djhankb: are you using LinuxBridge with vlan provider? | 20:39 |
| djhankb | The current one is 100% openvswitch | 20:40 |
| djhankb | which I think is a bit sadistic, as it introduced a ridiculous amount of challenges. | 20:41 |
| spatel | really? | 20:53 |
| spatel | i am using linuxBridge and thinking to use openvswitch | 20:53 |
| djhankb | well yeah, like on my controller nodes specifically. All of my containers are on OVS | 20:53 |
| spatel | We are running very CPU and network intensive application so currently cluster is running on all SR-IOV computes | 20:54 |
| djhankb | and there's some issues when you stop/start - you need to modify the attach script to run ovs-vsctl commands | 20:54 |
| djhankb | I think there's less complexity on the compute nodes. | 20:54 |
| spatel | Reason i need OVS because of dpdk :) sriov isn't fun (believe me) | 20:55 |
| djhankb | gotcha. | 20:55 |
| spatel | at some point i need to start thinking about how to convert linuxbridge to OVS | 20:55 |
| spatel | i am sure its not going to be fun | 20:55 |
| djhankb | yeah, I also had quite a bit of struggle getting a working /etc/network/interfaces file | 20:57 |
| djhankb | But - I always leave little notes and guides around to help my "future self" and others: https://gitlab.com/snippets/1941805 | 20:58 |
| spatel | nice!;... | 21:02 |
| spatel | i have to leave now.. but keep discussion going | 21:02 |
| spatel | i would love to understand all issue to deploy OVS :) | 21:02 |
| spatel | Thanks for your help | 21:02 |
| spatel | see you tomorrow | 21:02 |
| *** spatel has quit IRC | 21:03 | |
| *** DanyC has joined #openstack-ansible | 21:34 | |
| *** pcaruana has quit IRC | 21:36 | |
| CeeMac | djhankb: you're in about the same place as me. I didn't do ovs on controllers though. Did you have to hack that into the deployment config files or just ensure correctly named bridges exist? | 21:37 |
| *** ansmith has joined #openstack-ansible | 21:56 | |
| *** dave-mccowan has quit IRC | 22:05 | |
| *** spatel has joined #openstack-ansible | 22:12 | |
| *** spatel has quit IRC | 22:17 | |
| *** gundalow has quit IRC | 22:31 | |
| *** jrosser has quit IRC | 22:31 | |
| *** mwhahaha has quit IRC | 22:31 | |
| *** CeeMac has quit IRC | 22:31 | |
| *** xakaitetoia has quit IRC | 22:33 | |
| *** mwhahaha has joined #openstack-ansible | 22:33 | |
| *** gundalow has joined #openstack-ansible | 22:35 | |
| *** jrosser has joined #openstack-ansible | 22:36 | |
| *** guilhermesp has quit IRC | 22:36 | |
| *** mnaser has quit IRC | 22:36 | |
| *** strattao has quit IRC | 22:36 | |
| *** donnyd has quit IRC | 22:36 | |
| *** fyx has quit IRC | 22:36 | |
| *** gixx has quit IRC | 22:36 | |
| *** cyberpear has quit IRC | 22:36 | |
| *** fyx has joined #openstack-ansible | 22:38 | |
| *** mnaser has joined #openstack-ansible | 22:39 | |
| *** CeeMac has joined #openstack-ansible | 22:40 | |
| *** guilhermesp has joined #openstack-ansible | 22:41 | |
| *** gixx has joined #openstack-ansible | 22:42 | |
| *** donnyd has joined #openstack-ansible | 22:44 | |
| *** strattao has joined #openstack-ansible | 22:44 | |
| *** cyberpear has joined #openstack-ansible | 22:45 | |
| *** andrewbonney has quit IRC | 22:46 | |
| *** soren has quit IRC | 22:46 | |
| *** irclogbot_2 has quit IRC | 22:47 | |
| *** irclogbot_3 has joined #openstack-ansible | 22:47 | |
| *** soren has joined #openstack-ansible | 22:47 | |
| *** openstackstatus has quit IRC | 22:48 | |
| *** zbr has quit IRC | 22:50 | |
| *** mubix has quit IRC | 22:50 | |
| *** alanmeadows has quit IRC | 22:50 | |
| *** andrewbonney has joined #openstack-ansible | 22:52 | |
| *** alanmeadows has joined #openstack-ansible | 22:53 | |
| *** rh-jelabarre has quit IRC | 22:53 | |
| *** mubix has joined #openstack-ansible | 22:53 | |
| *** zbr has joined #openstack-ansible | 22:55 | |
| *** ianychoi has joined #openstack-ansible | 23:06 | |
| *** macz_ has quit IRC | 23:06 | |
| *** dave-mccowan has joined #openstack-ansible | 23:10 | |
| *** jbadiapa has quit IRC | 23:13 | |
| *** ahosam has quit IRC | 23:27 | |
| *** tosky has quit IRC | 23:44 | |
| *** KeithMnemonic has quit IRC | 23:51 | |
| *** KeithMnemonic has joined #openstack-ansible | 23:51 | |
| *** prometheanfire has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!