| dougszu | Has anyone noticed that OpenSearch services are restarted on consecutive service deploy/reconfigure runs with no change in config? | 10:25 |
|---|---|---|
| dougszu | I can look into it, but wondering if anyone else has started | 10:26 |
| Xedon | dougszu: the opensearch service is not working on my installation at all. I'm using the latest 2023.1 version. | 11:17 |
| dougszu | Xedon: What's the issue with it? If I had to guess, the container is in a restart loop? | 12:11 |
| Xedon | dougszu: the dashboard container yes. The other container is "unhealthy". I tried to open opensearch with curl. In the container I get an "unauthorized" and outside the container a connection reject. I am not sure what I am doing wrong. I actually only have "central logging" enabled. | 12:19 |
| dougszu | Xedon, for the opensearch_dashboards container, do the logs have something like `java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] `? | 12:24 |
| SvenKieske | you might be missing a patch which fixes dashboard permissions | 12:36 |
| SvenKieske | https://review.opendev.org/c/openstack/kolla-ansible/+/884649 | 12:37 |
| SvenKieske | make sure you have the appropriate backport of that installed. It's advised to install from the git branch, e.g. stable/zed; do NOT rely on pypi releases if you want to have a happy cloud deployment :) | 12:37 |
| Xedon | SvenKieske: ohh okay I see. I have the pypi release installed :D | 12:43 |
| SvenKieske | always use stable git branches please, the pypi release frequently miss bugfixes, as the releases are done manually and thus very infrequently. HTH | 13:00 |
| Xedon | SvenKieske: I installed the newest version with "pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2023.1". I re-deployed my lab after that, but I the opensearch container is still unhealthy... For example, I can see this warning: "Directory /etc/opensearch has insecure file permissions (should be 0700)" | 13:33 |
| Xedon | I also checked the "~/.local/share/kolla-ansible/ansible/roles/opensearch/templates/opensearch-dashboards.json.j2" file. It's the same as in the Repository https://opendev.org/openstack/kolla-ansible/commit/a3bbf3399bcff4e0089e189b2988c01da280650f | 13:35 |
| SvenKieske | mhm, maybe the fix was incomplete? | 13:36 |
| SvenKieske | can you maybe post your file permissions for that directory somewhere? https://paste.opendev.org/ | 13:37 |
| Xedon | SvenKieske: sure: https://paste.opendev.org/show/b853xDI5mOLeKwRaGokf/ | 13:41 |
| Xedon | SvenKieske: this are the warning messages https://paste.opendev.org/show/bC7trEauLd93b9osSQQi/ | 13:42 |
| SvenKieske | could you open a new bug on launchpad.net against kolla-ansible with this? also mention that you installed via pip|git; I'll be on vacation for almost 2 weeks; so someone else will need to take a look | 13:44 |
| SvenKieske | I see that the last patches didn't really touch on this directory permissions, wondering if upstream opensearch changed something | 13:45 |
| SvenKieske | you are the first user I'm hearing this from, weird. | 13:45 |
| SvenKieske | this might be related: https://github.com/opensearch-project/security/issues/1465 seems the security plugin yells at their own installer for not setting up correct permissions.. | 13:49 |
| SvenKieske | I guess I found maybe where a bug got introduced | 13:54 |
| SvenKieske | https://review.opendev.org/c/openstack/kolla-ansible/+/883942/4/ansible/roles/opensearch/templates/opensearch-dashboards.json.j2 | 13:55 |
| SvenKieske | I at least currently don't find anything else that ensured the permissions for /etc/opensearch/ this was the only place where that was set | 13:55 |
| SvenKieske | but that got reworked | 13:55 |
| SvenKieske | mhm no, that is just about the file permissions | 13:57 |
| SvenKieske | I guess the directory permissions are set by the opensearch installer, but I'm not 100% sure, need to investigate | 13:57 |
| Xedon | SvenKieske: I opened a bug report for that https://bugs.launchpad.net/kolla-ansible/+bug/2028376 | 14:02 |
| SvenKieske | ty | 14:04 |
| Xedon | SvenKieske: thank you for assistance! | 14:06 |
| SvenKieske | no problem | 14:09 |
| opendevreview | Doug Szumski proposed openstack/kolla-ansible master: Fix OpenSearch Dashboards health check https://review.opendev.org/c/openstack/kolla-ansible/+/889189 | 15:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!