| *** tosky has quit IRC | 00:02 | |
| *** klamath_atx has quit IRC | 00:14 | |
| *** macz_ has quit IRC | 00:19 | |
| *** klamath_atx has joined #openstack-ansible | 00:20 | |
| *** klamath_atx has quit IRC | 00:29 | |
| *** Adri2000 has quit IRC | 00:43 | |
| *** Adri2000 has joined #openstack-ansible | 00:44 | |
| *** jamesden_ has joined #openstack-ansible | 01:01 | |
| *** klamath_atx has joined #openstack-ansible | 01:01 | |
| *** jamesdenton has quit IRC | 01:03 | |
| *** spatel has joined #openstack-ansible | 01:51 | |
| *** spatel has quit IRC | 05:32 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** yasemind has joined #openstack-ansible | 05:42 | |
| *** pto has joined #openstack-ansible | 05:50 | |
| *** gyee has quit IRC | 05:59 | |
| *** openstackgerrit has joined #openstack-ansible | 06:08 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/ussuri: Python3 supported version of hatop https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/768739 | 06:08 |
|---|---|---|
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/ussuri: Python3 supported version of hatop https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/768739 | 06:12 |
| *** miloa has joined #openstack-ansible | 06:24 | |
| *** pcaruana has joined #openstack-ansible | 06:35 | |
| *** yasemind has quit IRC | 06:52 | |
| *** SiavashSardari has joined #openstack-ansible | 07:14 | |
| *** yasemind has joined #openstack-ansible | 07:16 | |
| *** luksky has joined #openstack-ansible | 07:23 | |
| *** ThiagoCMC has quit IRC | 07:34 | |
| yasemind | hi, i try to zun installation with openstack-ansible, when i run setup-openstack.yml, it is given an error http://paste.openstack.org/show/801757/. do you have any idea ? | 07:43 |
| *** jamesgibo has joined #openstack-ansible | 07:55 | |
| *** andrewbonney has joined #openstack-ansible | 08:09 | |
| *** maharg101 has joined #openstack-ansible | 08:13 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Remove note about metal/horizon compatability https://review.opendev.org/c/openstack/openstack-ansible/+/771573 | 08:25 |
| noonedeadpunk | yasemind: what release are you trying to run? | 08:33 |
| noonedeadpunk | also andrewbonney right now is doing a lot of stuff related to zun role, so I think you could try using master version of the role | 08:34 |
| yasemind | noonedeadpunk i use stable/ussuri | 08:34 |
| noonedeadpunk | it would still install zun from ussuri though | 08:34 |
| noonedeadpunk | or at least you can try to check out role to stable/victoria | 08:35 |
| noonedeadpunk | cd /etc/ansible/roles/os_zun; git checkout origin/stable/victoria | 08:35 |
| yasemind | our system is stable/ussuri so i must install stable/ussuri | 08:36 |
| noonedeadpunk | Actually, roles can install any version of zun service itself. So even using victoria role you would still have the correct ussuri version of the service. Since all roles rely on versions provided in https://opendev.org/openstack/openstack-ansible/src/branch/stable/ussuri/playbooks/defaults/repo_packages/openstack_services.yml#L326-L330 | 08:38 |
| noonedeadpunk | I think zun role is broken on Ussuri and it needs a lot of cherry-picking | 08:41 |
| *** mgariepy has quit IRC | 08:41 | |
| *** mgariepy has joined #openstack-ansible | 08:41 | |
| yasemind | i try to git checkout origin/stable/victoria, but it gave an error error: pathspec 'origin/stable/victoria' did not match any file(s) known to git | 08:42 |
| noonedeadpunk | try doing git pull or git fetch first | 08:43 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun stable/ussuri: Update zun role to match current requirements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/771547 | 08:43 |
| jrosser | andrewbonney: what is the state of zun on victoria branch, should we be continuing to backport your current patches? | 08:44 |
| andrewbonney | It's in a working state. Anything on master now which hasn't been cherry picked is more of a feature or tidy up | 08:44 |
| andrewbonney | Backporting to ussuri ought to work if we want, but we might hit a CI issue with kuryr as it will need a backport to ussuri too | 08:45 |
| *** akahat|rover is now known as akahat|lunch | 08:46 | |
| *** ahsen has joined #openstack-ansible | 08:47 | |
| *** tosky has joined #openstack-ansible | 08:47 | |
| noonedeadpunk | On U we don't have tempest as well I guess... | 08:47 |
| noonedeadpunk | (for zun) | 08:47 |
| noonedeadpunk | so if role is working - CI will just pass. But yeah, I guess we might want to backport kuryr as well | 08:48 |
| *** ahsen has quit IRC | 08:48 | |
| andrewbonney | Not sure. I don't remember changing tempest but could be wrong | 08:48 |
| *** ahsen has joined #openstack-ansible | 08:48 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client master: Remove variable examples which are already deprecated https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/771575 | 08:49 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Disable repeatedly failing zun tempest test https://review.opendev.org/c/openstack/openstack-ansible/+/771548 | 08:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun stable/ussuri: Update zun role to match current requirements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/771547 | 08:57 |
| *** spatel has joined #openstack-ansible | 09:09 | |
| *** luksky has quit IRC | 09:12 | |
| *** luksky has joined #openstack-ansible | 09:13 | |
| *** spatel has quit IRC | 09:14 | |
| *** raukadah has quit IRC | 09:18 | |
| *** raukadah has joined #openstack-ansible | 09:20 | |
| manti | When running setup-openstack.yml, at os_keystone : Create database for service task, I get error "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required.". I can see the task is delegated to utility container, so does the error come from utility container or is there some other task delegation happening to some other container? | 09:25 |
| manti | Also, when is PyMySQL installed? I have run the playbooks with verbose but I'm not seeing anything related to PyMySQL, so I'm again lost at what phase something is not working | 09:28 |
| noonedeadpunk | yes, all mysql related tasks are delegated to utility container. And pymysql should be performed during utility-install playbook https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/utility_all.yml#L58 | 09:37 |
| noonedeadpunk | this should be part of setup-infrastructure.yml | 09:38 |
| noonedeadpunk | jrosser: I literally just saw that override https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/utility_all.yml#L91 | 09:39 |
| noonedeadpunk | Wondering if we need it? | 09:39 |
| jrosser | well it looks odd having it there, and its also overriding this https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L98 | 09:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon stable/victoria: Add missing 'horizon-config' tag https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/771549 | 09:59 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon stable/ussuri: Add missing 'horizon-config' tag https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/771550 | 09:59 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon stable/ussuri: Add missing 'horizon-config' tag https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/771550 | 10:00 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client master: Remove examples for variables which no longer exist https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/771575 | 10:03 |
| tacco | jrosser: i guess this could be the problem for my ssl proxy issues "#enable_proxy_headers_parsing = false | 10:03 |
| tacco | default seems to be false. :> | 10:03 |
| *** SiavashSardari has quit IRC | 10:04 | |
| jrosser | tacco: have you tried changing that to see if it helps? | 10:07 |
| *** akahat|lunch is now known as akahat|rover | 10:09 | |
| tacco | will do in a second. :) | 10:09 |
| *** SiavashSardari has joined #openstack-ansible | 10:10 | |
| *** SiavashSardari has joined #openstack-ansible | 10:11 | |
| tacco | ok tested.. when i enable the header-parsing i get a url like the following | 10:18 |
| tacco | {"versions":{"values":[{"id":"v2","links":[{"href":"https,https://cloud.os.example.net:9001/v2","rel":"self"}],"status":"CURRENT"}]}} | 10:18 |
| tacco | https,https seems to be still a bit buggy but the header now seems to be recognized. | 10:18 |
| tacco | now i only have to get rid of the https, in front of the url and everything shouls be fine. :) | 10:19 |
| *** rpittau is now known as rpittau|bbl | 10:20 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/767902 | 10:23 |
| *** priteau has joined #openstack-ansible | 10:24 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client stable/victoria: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/767914 | 10:24 |
| *** SiavashSardari has quit IRC | 10:26 | |
| *** SiavashSardari has joined #openstack-ansible | 10:26 | |
| *** yasemind has quit IRC | 10:26 | |
| *** yasemind has joined #openstack-ansible | 10:28 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Revert "Roll back use of tempest smoke test for the integrated repo" https://review.opendev.org/c/openstack/openstack-ansible/+/771553 | 10:28 |
| *** shyamb has joined #openstack-ansible | 10:34 | |
| tacco | jrosser: ok the bug with the https,https://... was because of setting both x-proto.. and x_proto as header.. removed that "old fasioned" version x_proto and everything works like a charm. | 10:38 |
| jrosser | tacco: it would be great if you could create a bug about this on launchpad | 10:38 |
| tacco | i persisted this change in osa by user_designate_proxy_header_fix.yml with designate_designate_conf_overrides: | 10:39 |
| tacco | there was already one https://bugs.launchpad.net/openstack-ansible/+bug/1713663 | 10:39 |
| openstack | Launchpad bug 1713663 in openstack-ansible "Set enable_proxy_headers_parsing = True when HAProxy is used" [Medium,Fix released] - Assigned to Shannon Mitchell (shannon-mitchell) | 10:39 |
| tacco | https://opendev.org/openstack/openstack-ansible-os_designate/commit/7cdb8ea27c017fd056edfea056114b5329f8248a | 10:40 |
| tacco | not sure why i don't have this fix already but with the custom config it works as well. | 10:41 |
| jrosser | but theres also something with the headers though? | 10:41 |
| tacco | no the header thing was my fault. | 10:41 |
| jrosser | which branch are you using? | 10:41 |
| tacco | because i set them twice yesterday for testing because i wans't sure which one to use. | 10:41 |
| tacco | stable/train | 10:41 |
| tacco | yes yes yes.. have to upgrade. :> | 10:42 |
| SiavashSardari | hey everyone. I have a question regarding the os-gnocchi-install playbook. to have gnocchi work with ceph we need the ceph_client role install the gnocchi user keyring in gnocchi container. but ceph_client droped gnocchi integration. should I add ceph_extra_components to my user_vars os pass that variable in os-gnocchi-install playbook? | 10:42 |
| *** shyamb has quit IRC | 10:42 | |
| SiavashSardari | and why ceph_client droped gnocchi integration? | 10:43 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:43 | |
| *** shyamb has joined #openstack-ansible | 10:51 | |
| jrosser | SiavashSardari: i'm not sure what you mean about ceph_client dropping gnocchi integration, there is documentation here https://github.com/openstack/openstack-ansible-ceph_client/blob/master/doc/source/configure-ceph.rst | 10:56 |
| SiavashSardari | jrosser Thanks for the link. | 11:04 |
| SiavashSardari | if you take a look at https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/vars/main.yml#L47-L55 you can see that it says ceph_client droped gnocchi integration | 11:05 |
| SiavashSardari | it also kinda says that here https://github.com/openstack/openstack-ansible-ceph_client/blob/master/doc/source/configure-ceph.rst#configure-os_gnocchi-with-ceph_client | 11:06 |
| SiavashSardari | I am working on using gnocchi with ceph and redis and I think the documentation in osa is out dated. | 11:08 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Run linters job on a focal node instead of bionic https://review.opendev.org/c/openstack/openstack-ansible-tests/+/771498 | 11:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Added tag neutron-policy-overrides to just update neutron policy https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/770248 | 11:22 |
| *** shyamb has quit IRC | 11:23 | |
| *** rpittau|bbl is now known as rpittau | 11:24 | |
| *** macz_ has joined #openstack-ansible | 11:34 | |
| SiavashSardari | jrosser about policy migration from json to yaml, I think just removing json file and adding a new yaml file would be suffice. I don't see any scenario which requires loading existing json file to yaml format. what is your opinion about it? | 11:35 |
| *** macz_ has quit IRC | 11:39 | |
| *** ahsen has quit IRC | 12:03 | |
| yasemind | noonedeadpunk and jrosser we try to https://opendev.org/openstack/openstack-ansible/src/branch/stable/victoria/playbooks/defaults/repo_packages/openstack_services.yml#L292-L305 and we git checkout stable/victoria branch, but it gave an different error : http://paste.openstack.org/show/801765/ | 12:07 |
| jrosser | SiavashSardari: there is some discussion here http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019910.html | 12:07 |
| jrosser | yasemind: after you change branch on openstack-ansible you must re-run bootstrap-ansible.sh | 12:08 |
| jrosser | it looks like the ansible collection for rabbitmq is missing "ERROR! couldn't resolve module/action 'community.rabbitmq.rabbitmq_vhost'" | 12:09 |
| jrosser | this is installed during the ansible boostrap script | 12:09 |
| noonedeadpunk | uh | 12:16 |
| noonedeadpunk | the idea there was to use U while using V for specific role | 12:16 |
| noonedeadpunk | so bootstrap-ansible.sh is not an option here... | 12:17 |
| jrosser | oh - right | 12:17 |
| jrosser | user_role_requirements then? | 12:18 |
| noonedeadpunk | yasemind: sorry I completely forgot about that nuance. I think on U you still can install collections though. you need to add these 3 collections to your file https://opendev.org/openstack/openstack-ansible/src/branch/master/ansible-collection-requirements.yml#L9-L17 | 12:18 |
| noonedeadpunk | or into random file and run /opt/ansible-runtime/bin/ansible-galaxy collection install -r ansible-collection-requirements.yml -p /etc/ansible | 12:22 |
| noonedeadpunk | but maybe it's worth testing our backports to U | 12:22 |
| *** yasemind has quit IRC | 12:24 | |
| *** yasemind has joined #openstack-ansible | 12:27 | |
| *** SiavashSardari has quit IRC | 12:28 | |
| *** SiavashSardari has joined #openstack-ansible | 12:28 | |
| SiavashSardari | jrosser thanks for the discussion link, but I got little bit confused. To my understanding default values of policy rules are handled in code unless some rule is overwritten via a policy file (either json or yaml, but not both) | 12:32 |
| SiavashSardari | so it doesn't matter if we have an old json file. while the config is referring to a yaml file | 12:33 |
| SiavashSardari | I think I misunderstood something here. | 12:34 |
| yasemind | noonedeadpunk i try to your command /opt/ansible-runtime/..... give an error : ERROR! Collection artifact at '/root/.ansible/tmp/ansible-local-73673p_9fimyx/tmpdliq6myz/communityhjm76ok2.general' is not a valid tar file. | 12:36 |
| yasemind | any idea ? | 12:36 |
| noonedeadpunk | doh, yes.... | 12:37 |
| noonedeadpunk | can replace colelctions I provided you with http://paste.openstack.org/show/801766/ ? | 12:38 |
| *** pcaruana has quit IRC | 12:43 | |
| *** ahsen has joined #openstack-ansible | 12:50 | |
| yasemind | noonedeadpunk it didnt working; give an error http://paste.openstack.org/show/801768/ but, i run ansible-galaxy collection install community.rabbitmq, it works | 12:51 |
| *** evrardjp_ has joined #openstack-ansible | 12:52 | |
| noonedeadpunk | and you need to install mysql as well | 12:52 |
| noonedeadpunk | and can avoid crypto I guess | 12:53 |
| *** rh-jelabarre has joined #openstack-ansible | 12:54 | |
| *** evrardjp has quit IRC | 12:54 | |
| *** pcaruana has joined #openstack-ansible | 12:55 | |
| *** rh-jelabarre has quit IRC | 13:01 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:04 | |
| *** spatel has joined #openstack-ansible | 13:10 | |
| *** rh-jelabarre has quit IRC | 13:11 | |
| *** spatel has quit IRC | 13:15 | |
| admin0 | has anyone have ipv6 with custom host-route ? | 13:18 |
| admin0 | in my case, it does not seem to work at all | 13:18 |
| admin0 | i added: Additional Routes Destination ::/0 : Next hop xxxx:1a00:8011::1 .. but insdie the instance, i see it adding route to some fe80::232:17ff:fe74:d024 as gateway | 13:19 |
| *** rh-jelabarre has joined #openstack-ansible | 13:22 | |
| *** rh-jelabarre has quit IRC | 13:22 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:22 | |
| yasemind | noonedeadpunk i run setup script, but failed: [infra1_zun_api_container-237ad141 -> localhost] (item={'name': 'zun', 'users': [{'username': 'zun', 'password': '86e22bfcbdb3a340504f4'}]}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zun", "users": [{"password": "xxxxxx", "username": "zun"}]}, "msg": "The PyMySQL (Python 2.7 | 13:30 |
| yasemind | and Python 3.X) or MySQL-python (Python 2.X) module is required."} | 13:30 |
| yasemind | it checked the containers, but it is installed, it didint understand it. any idea ? | 13:30 |
| *** dasp_ has joined #openstack-ansible | 13:31 | |
| *** dasp has quit IRC | 13:34 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update pip/setuptools/wheel to latest version https://review.opendev.org/c/openstack/openstack-ansible/+/770284 | 13:51 |
| *** spatel has joined #openstack-ansible | 14:01 | |
| *** ahsen has quit IRC | 14:03 | |
| *** SiavashSardari has quit IRC | 14:04 | |
| *** yasemind has quit IRC | 14:04 | |
| *** yasemind has joined #openstack-ansible | 14:04 | |
| *** yasemind has quit IRC | 14:08 | |
| *** SiavashSardari has joined #openstack-ansible | 14:09 | |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Bring db setup vars in line with other roles https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/771652 | 14:15 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update pip/setuptools/wheel to latest version https://review.opendev.org/c/openstack/openstack-ansible/+/770284 | 14:19 |
| *** pto has quit IRC | 14:23 | |
| *** irclogbot_0 has quit IRC | 14:27 | |
| *** irclogbot_2 has joined #openstack-ansible | 14:30 | |
| *** rh-jelabarre has quit IRC | 14:38 | |
| *** ThiagoCMC has joined #openstack-ansible | 15:01 | |
| *** d34dh0r53 has quit IRC | 15:12 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Change format of rows https://review.opendev.org/c/openstack/openstack-ansible/+/771659 | 15:15 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Change format of rows https://review.opendev.org/c/openstack/openstack-ansible/+/771659 | 15:16 |
| *** d34dh0r53 has joined #openstack-ansible | 15:19 | |
| *** b1tsh1ft3r has joined #openstack-ansible | 15:26 | |
| jrosser | jamesden_: any thooughts on this, and the proposed solution? https://bugs.launchpad.net/openstack-ansible/+bug/1846198 | 15:54 |
| openstack | Launchpad bug 1846198 in openstack-ansible " packet loss during active L3 HA agent restart" [Undecided,New] - Assigned to James Denton (james-denton) | 15:54 |
| mgariepy | cant we just set a override for the service ? | 16:01 |
| mgariepy | for the killmode? | 16:02 |
| noonedeadpunk | we totally can | 16:02 |
| jrosser | we were just testing minor upgrades in the lab and very surprised at total outage for l3 agent stuff for a long time | 16:03 |
| mgariepy | neutron is such a pain. | 16:03 |
| *** macz_ has joined #openstack-ansible | 16:05 | |
| mgariepy | https://github.com/rdo-packages/neutron-distgit/blob/rpm-master/neutron-l3-agent.service | 16:14 |
| jrosser | unfortunately it is ugly, becasue it seems to leave privsep-helper and neutron-keepalived-state-change also running | 16:15 |
| jrosser | so if it's an upgrade, youve got those from the old venv | 16:15 |
| *** theintern_ has joined #openstack-ansible | 16:21 | |
| *** b1tsh1ft3r has quit IRC | 16:22 | |
| *** jamesgibo has quit IRC | 16:34 | |
| noonedeadpunk | `Note that it is not recommended to set KillMode= to process or even none, as this allows processes to escape the service manager's lifecycle and resource management` | 16:35 |
| noonedeadpunk | https://www.freedesktop.org/software/systemd/man/systemd.kill.html | 16:35 |
| *** gyee has joined #openstack-ansible | 16:50 | |
| mgariepy | is there a way to set the agent keepalived to slave before the restart? | 16:52 |
| mgariepy | moving the active ips from the current to the other agent server would reduce the downtime | 16:52 |
| noonedeadpunk | I think it's done by default for routers that have ha. The thing is that we might restart another network node before l3 agent is fully started | 16:53 |
| noonedeadpunk | once keepalived is killed, active ip must move to slave | 16:55 |
| noonedeadpunk | for it it was almost instant process. but when both master and slave are dead... that's an issue... | 16:56 |
| mgariepy | is there a lot of ppl running in both ha and non-ha at the same time? | 17:02 |
| *** evrardjp has joined #openstack-ansible | 17:03 | |
| jrosser | mgariepy: i think the point is there isnt a need for any downtime, the neutron code expects to be able to restart the l3 agent without touching the keepalived processes | 17:04 |
| jrosser | this is however at odds with the way systemd sees the world, and as it stands today when we restart the l3 agent at upgrade, all the keepalived are also killed unnecessarily | 17:05 |
| *** evrardjp_ has quit IRC | 17:06 | |
| *** jamesgibo has joined #openstack-ansible | 17:14 | |
| *** dotnetted has joined #openstack-ansible | 17:16 | |
| *** miloa has quit IRC | 17:20 | |
| dotnetted | Let me know if support questions are not allowed here. Using the current master branch, installing onto an Ubuntu 20.04 target, task [os_keystone : Create database for service] fails with "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required.". This is caused by a ModuleNotFound exception during "import pymysql". The target host has python3-pymysql installed. How does the | 17:23 |
| dotnetted | virtualenv used by openstack-ansible populate its modules? - Thanks | 17:23 |
| *** evrardjp_ has joined #openstack-ansible | 17:25 | |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| *** evrardjp has quit IRC | 17:28 | |
| *** SiavashSardari has quit IRC | 17:32 | |
| noonedeadpunk | dotnetted: this is delegated to utility container which should have this installed | 17:35 |
| noonedeadpunk | eventually inside utility virtualenv | 17:35 |
| noonedeadpunk | I see this issue too often for the last 2 days - maybe we made some wrong assumption somewhere? | 17:36 |
| dotnetted | I see - Thanks | 17:36 |
| *** theintern_ has quit IRC | 17:38 | |
| dotnetted | Also, probably unrelated but just in case: I tried installing via Kayobe before attempting openstack-ansible and I came across a slightly similar issue. Kayobe master requires "MichaelRigart.interfaces v1.9.2" which has the wrong package name for python3-selinux. (Needs updating from python-selinux) | 17:38 |
| dotnetted | Possibly this is in some way similar, trying to use python-pymysql instead of python3-pymysql somewhere. | 17:39 |
| jrosser | dotnetted: can you paste the actual output from ansible for that task to paste.openstack.org? | 17:40 |
| dotnetted | Yep will do. | 17:40 |
| jrosser | did you have any errors before this point, during setup-infrastructure.yml ? | 17:40 |
| dotnetted | It also occurs on "[galera_server : Create galera users]" - I assume it's whichever task is reached first. | 17:41 |
| dotnetted | I had to disable no_log in the related yaml files to see the actual error. | 17:41 |
| dotnetted | Apart from these 2 issues I havnt seen any other errors with openstack-ansible so far | 17:43 |
| noonedeadpunk | [galera_server : Create galera users] - looks alerting | 17:43 |
| noonedeadpunk | I think we see things being stuck here in CI from time to time | 17:44 |
| noonedeadpunk | but re-running of the role should fix issue | 17:44 |
| dotnetted | I'm running it again now to get the exact output, but here's a snippet: http://paste.openstack.org/show/801785/ | 17:44 |
| noonedeadpunk | have you finished galera role in success state at the end? | 17:44 |
| dotnetted | Yes, apart from when that one randomly errors. | 17:45 |
| noonedeadpunk | can you re-run openstack-ansible playbooks/utility-install.yml ? | 17:46 |
| dotnetted | I put some debugging into /etc/ansible/ansible_collections/community/mysql/plugins/module_utils/mysql.py and verified the "import pymysql" line is throwing a ModuleNotFound" | 17:46 |
| jrosser | you should see the task delegated to the utility container | 17:46 |
| noonedeadpunk | you digged too deep :) | 17:46 |
| jrosser | and the python interpreter should set to the utility venv in /openstack/venv | 17:47 |
| jrosser | i have seen something wierd with galera on focal AIO, and it looked like the galera process did not start properly | 17:47 |
| noonedeadpunk | we should do smth to galera... but having no idea what's wrong there... | 17:47 |
| noonedeadpunk | it's some race condition | 17:48 |
| *** evrardjp_ has quit IRC | 17:48 | |
| jrosser | restarting galera by hand and checking the journal that its running properly and not in some wierd state let me carry on | 17:48 |
| jrosser | when it was broken the journal really didnt have much "success" in it | 17:48 |
| dotnetted | utility-install.yml ok=50 skipped=17 | 17:48 |
| noonedeadpunk | I just re-ran role... | 17:48 |
| jrosser | also i noticed that galera can sometimes take really very long to restart | 17:49 |
| jrosser | i was wondering if ansible is giving it long enough | 17:49 |
| noonedeadpunk | can you check that in /openstack/venvs/utility-xx.xx.x/bin/pip list | grep -i pymysql there's a module installed? | 17:49 |
| jrosser | or rather, service start -> service usable is long | 17:49 |
| noonedeadpunk | and we have a wait timeout only in CI tests? | 17:50 |
| noonedeadpunk | that might be the case... I think we don't verify that servce in usavble state before proceeding | 17:50 |
| jrosser | like you i see focal CI errors a bit like this | 17:50 |
| dotnetted | silly question heh: where exactly is /openstack | 17:50 |
| dotnetted | how can I access the container ( I presume) | 17:50 |
| dotnetted | oh nevermind... I swear it wasn't there before lol | 17:51 |
| noonedeadpunk | on controller node you can lxc-ls to list containers and lxc-attach -n <container_name> to attach into utility container | 17:51 |
| noonedeadpunk | and there you should be able to find venvs | 17:51 |
| *** evrardjp has joined #openstack-ansible | 17:51 | |
| noonedeadpunk | if module is not there -try running openstack-ansible playbooks/utility-install.yml -e venv_rebuild=true | 17:52 |
| dotnetted | http://paste.openstack.org/show/801786/ | 17:52 |
| noonedeadpunk | sorry, need to run away to the shop until they're open.... which soon won't be the case :( | 17:52 |
| spatel | admin0: around? | 17:53 |
| dotnetted | Running utility-install w/ rebuild now | 17:53 |
| * admin0 is around | 17:53 | |
| admin0 | my 15th cluster on OSA on 12th country is up :) | 17:53 |
| noonedeadpunk | o_O | 17:53 |
| admin0 | office + upwork | 17:54 |
| spatel | admin0: are you using pdns for designate or bind? | 17:54 |
| admin0 | bind | 17:54 |
| admin0 | i tried pdns .. could not move anywhere :( | 17:54 |
| admin0 | someone here has done is using pds | 17:54 |
| spatel | where do you running mDNS service? | 17:54 |
| noonedeadpunk | I made pdns work actually | 17:54 |
| admin0 | i jus forgot his name | 17:54 |
| admin0 | oh :) | 17:54 |
| admin0 | right in front of me :) | 17:55 |
| admin0 | sorry | 17:55 |
| admin0 | mDNS .. what is that ? i just run the playbooks with right variables and it works :) | 17:55 |
| spatel | noonedeadpunk: Can PowerDNS run in two mode master and slave together? | 17:55 |
| admin0 | the desginate container pushes the records to the bind masters | 17:55 |
| spatel | mDNS is python based DNS which designate container run | 17:55 |
| admin0 | as slaves, but they are master for the world | 17:56 |
| noonedeadpunk | yes, I have master/slave option, attached to mysql | 17:56 |
| admin0 | to be frank, i only do journalctl --follow inside the desigate container .. i have not done a ps | 17:56 |
| spatel | I do have master/slave but i want single machine act for slave for A domain and Master for B domain | 17:56 |
| noonedeadpunk | so you write to single master, and it is synced to the salve via api (if i'm not mixing how this works with bind) | 17:56 |
| admin0 | spatel, for us, we treat the slave( the dns server ) as hidden master .. and then we sync the records to other servers which act as master | 17:57 |
| admin0 | so designate will push all to your dns (hidden master) ... from there, you can sync to the real ones | 17:58 |
| spatel | Yes designate act as master and you can replicate them to slave | 17:58 |
| admin0 | i meant, we have this process designate -> (bind9 , hidden master) -> real masters ( public IPs) | 17:59 |
| admin0 | in this case, we can decide which domain syncs where | 17:59 |
| noonedeadpunk | that was my setup http://paste.openstack.org/show/801787/ | 17:59 |
| spatel | I am looking for solution where my pDNS slave also act like master for some domain (i am not seeing that kind of configuration in PDNS where it support both roles) | 17:59 |
| noonedeadpunk | considering you have an extra interface attached to the designate containers with public network that it will use to talk to external pdns servers | 18:00 |
| noonedeadpunk | and you have pdns cluster set up as well | 18:00 |
| noonedeadpunk | I think it would be hard to achieve since you need also to tel designate what domain write to what instance... | 18:01 |
| spatel | noonedeadpunk: thank for sharing that config. so ns1 and ns2 are slave pDNS ? | 18:02 |
| noonedeadpunk | here might be the easiest thing is to have several powerdns clusters and configure them independently... | 18:02 |
| noonedeadpunk | ns1.cloud.example.com resolves to xx.xx.xx.xx and ns2.cloud.example.com to yy.yy.yy.yy | 18:02 |
| spatel | hmm | 18:03 |
| noonedeadpunk | so for designate they are both masters, but it writes only to the one provided in target | 18:04 |
| noonedeadpunk | * target options | 18:04 |
| spatel | oh okay | 18:04 |
| spatel | let me try something | 18:04 |
| spatel | noonedeadpunk: side question did we release RC2 of victoria? | 18:05 |
| noonedeadpunk | actually, designate mdns is kind of master here :)) it just closed from the world, so pwoerdns is facing it , caching and forwarding requests to mdns when needed iirc | 18:05 |
| noonedeadpunk | I think we did | 18:05 |
| noonedeadpunk | rc2 == 22.0.0 | 18:05 |
| noonedeadpunk | btw talking about this, mgariepy, if you're around, can you vote on https://review.opendev.org/c/openstack/openstack-ansible/+/771148 ? | 18:06 |
| * noonedeadpunk runs away to the shop | 18:07 | |
| *** evrardjp_ has joined #openstack-ansible | 18:19 | |
| *** cloudnull has quit IRC | 18:19 | |
| *** cloudnull has joined #openstack-ansible | 18:20 | |
| *** evrardjp has quit IRC | 18:22 | |
| *** cloudnull5 has joined #openstack-ansible | 18:26 | |
| *** cloudnull has quit IRC | 18:27 | |
| *** cloudnull5 is now known as cloudnull | 18:27 | |
| *** evrardjp has joined #openstack-ansible | 18:32 | |
| *** evrardjp_ has quit IRC | 18:35 | |
| mgariepy | noonedeadpunk, dong. sorry i was lunching.. | 18:37 |
| mgariepy | done** | 18:37 |
| mgariepy | i wonder if we could just simply create a slice for the l3-agent subprocess to put them here. | 18:39 |
| *** andrewbonney has quit IRC | 19:09 | |
| *** b1tsh1ft3r has joined #openstack-ansible | 19:29 | |
| *** jamesgibo has quit IRC | 19:39 | |
| *** b1tsh1ft3r has quit IRC | 19:46 | |
| *** maharg101 has quit IRC | 20:06 | |
| dotnetted | noonedeadpunk, thanks for pointing me in the right direction, turned out to be haproxy binding silently failing causing utility install to fail package installation, which in turn caused the pymysql package 404 | 20:37 |
| *** dave-mccowan has joined #openstack-ansible | 20:40 | |
| *** tosky has quit IRC | 20:41 | |
| *** tosky has joined #openstack-ansible | 20:42 | |
| *** raukadah has quit IRC | 20:42 | |
| *** mgariepy has quit IRC | 20:42 | |
| *** raukadah has joined #openstack-ansible | 20:43 | |
| *** spatel has quit IRC | 20:43 | |
| *** mgariepy has joined #openstack-ansible | 20:43 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/victoria: Add missing 'horizon-config' tag https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/771549 | 21:01 |
| jrosser | dotnetted: nice to hear its working, let us know if you get stuck again | 21:03 |
| *** jamesden_ has quit IRC | 21:51 | |
| *** jamesdenton has joined #openstack-ansible | 21:51 | |
| *** maharg101 has joined #openstack-ansible | 22:03 | |
| *** maharg101 has quit IRC | 22:07 | |
| *** cloudnull8 has joined #openstack-ansible | 22:44 | |
| dotnetted | Well now that pymysql is available it is able to attempt the connection to MySQL, which fails horribly lol. What's the best way to debug why haproxy is having problems sending requests through to galera? | 22:44 |
| dotnetted | http://paste.openstack.org/show/801803/ | 22:44 |
| dotnetted | I can access galera from the host w/out issue at the same IP/Port set in the haproxy backend - connecting to the haproxy port gives the following: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2 (which is understandably killing the install) | 22:45 |
| *** cloudnull has quit IRC | 22:46 | |
| *** cloudnull8 is now known as cloudnull | 22:46 | |
| dotnetted | Task Output: http://paste.openstack.org/show/801804/ | 22:48 |
| jrosser | dotnetted: have you build an all-in-one first? | 23:05 |
| dotnetted | It's technically an all-in-one configuration right now: http://paste.openstack.org/show/801805/ | 23:06 |
| dotnetted | (I think) | 23:07 |
| dotnetted | With a fairly simple netplan: http://paste.openstack.org/show/801806/ | 23:08 |
| jrosser | 10.0.60.15 should probably also be in used_ips (not that this is related to your issue) | 23:10 |
| dotnetted | ah yeah thanks :) - It doesn't actually exist right now heh - was going to deal with that section after getting it up & running | 23:11 |
| jrosser | 9k mtu on bt-mgmt is likley going to be a problem | 23:11 |
| jrosser | if you look inside the lxc eth1 is almost certainly 1500 mtu | 23:12 |
| jrosser | if you can spin up an 8 core 8G ram 60G disk VM somewhere and run the vanilla AIO deployment you'll get a fairly instant reference point to compare with | 23:13 |
| jrosser | the exact same AIO config is run 100's of times a day in CI | 23:13 |
| dotnetted | Ok thanks - will do | 23:14 |
| jrosser | tbh "smells like network trouble" from what you've pasted | 23:14 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Bump SHAs for stable/ussuri https://review.opendev.org/c/openstack/openstack-ansible/+/771148 | 23:14 |
| dotnetted | Its just weird that haproxy running on the host cant contact the backend but I can connect directly from the host to the same IP. | 23:15 |
| jrosser | perhaps that depends on your source ip | 23:16 |
| jrosser | if you've got some other physical interface with a 1500 mtu and the default route, then that may be used for your testing by hand | 23:17 |
| dotnetted | im gonna drop all the mtus to 1500 and see what happens | 23:17 |
| jrosser | haproxy is bound to the 9k mtu bt-mgmt ip | 23:17 |
| jrosser | just the br-mgmt, the others are fine | 23:18 |
| jrosser | ideally tunnel needs to be > 1500 for vxlan headers | 23:18 |
| jrosser | anyway its late here, good luck :) | 23:19 |
| dotnetted | Thanks! | 23:21 |
| *** luksky has quit IRC | 23:33 | |
| *** tosky has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!