| jrosser | good morning | 08:28 |
|---|---|---|
| damiandabrowski | hi! | 08:40 |
| noonedeadpunk | o/ | 08:52 |
| jrosser | i think rgw creates its own pools if they are not present | 08:52 |
| jrosser | https://docs.ceph.com/en/reef/radosgw/pools/#tuning | 08:53 |
| jrosser | so that is not the issue | 08:53 |
| noonedeadpunk | but what about users then? | 09:04 |
| noonedeadpunk | or there shouldn't be requirements, except providing correct keystone auth | 09:05 |
| * noonedeadpunk can't recall already | 09:05 | |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Override pdf filename for the deploy guide https://review.opendev.org/c/openstack/openstack-ansible/+/922130 | 09:13 |
| opendevreview | Gaudenz Steinlin proposed openstack/openstack-ansible-galera_server master: Fix condition in cluster state checks https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922375 | 09:16 |
| jrosser | i cant recall either tbh | 09:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update pip/wheel/setuptools/uwsgi versions to latest https://review.opendev.org/c/openstack/openstack-ansible/+/922376 | 09:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.11.8 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922377 | 09:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Update rabbitmq-server to 3.13.3 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/922378 | 09:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: [doc] Override pdf filename for the deploy guide https://review.opendev.org/c/openstack/openstack-ansible/+/922380 | 09:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2024.1: [doc] Override pdf filename for the deploy guide https://review.opendev.org/c/openstack/openstack-ansible/+/922380 | 09:55 |
| opendevreview | Gaudenz Steinlin proposed openstack/openstack-ansible-galera_server master: Fix condition in cluster state checks https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922375 | 10:00 |
| noonedeadpunk | jrosser: I think we should try mariadb 11.4.2 instead | 10:10 |
| noonedeadpunk | It's a new LTS and it's out of RC now | 10:11 |
| noonedeadpunk | (still missing 24.04 repos though) | 10:11 |
| gaudenz_ | noonedeadpunk: Regarding your comment in https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922375 I'm not sure I agree with you on the your interpretation of https://galeracluster.com/library/documentation/node-states.html#node-state-changes | 10:19 |
| gaudenz_ | I can observe that 4 means "Synced", so if the numbers in the diagram correspond to states, they belong to the state where the arrow is pointing, but then "Open" has no number and "Joined" has 3 and 6? | 10:21 |
| gaudenz_ | My interpretation would be that this is just a reference to the steps in the text below and has nothing to do with the values in "wsrep_local_state". | 10:22 |
| noonedeadpunk | Well, I somehow can recall that Donor is still 5, and Joiner 2. And also you do change logic there, as Joiner was also considered as "ok" previously | 10:23 |
| noonedeadpunk | at very least I think it's worth to be 2 independent changes, to vote on them separately | 10:24 |
| noonedeadpunk | as also - it might be worth checking for `wsrep_ready` as well... | 10:24 |
| gaudenz_ | OK, I can split them. | 10:24 |
| gaudenz_ | But still I would have to change the "in" to "not in", otherwise it fails if the cluster is OK. This was previously masked by the first condition. | 10:25 |
| noonedeadpunk | though you might be right in terms of != 4 | 10:27 |
| noonedeadpunk | yeah, but you can push second patch on top of the first one not to fight with merge conflicts | 10:27 |
| noonedeadpunk | as I see this logic issue totally worth backporting, while I'm not sure if change to wsrep_local_state_comment should be backported as well | 10:28 |
| noonedeadpunk | was checking for proxysql docs, and they do check wsrep_local_state there just in case | 10:29 |
| noonedeadpunk | https://proxysql.com/documentation/galera-configuration/ | 10:29 |
| gaudenz_ | I think wsrep_local_state == 4 and wsrep_local_state_comment == "Synced" are the same. And this is also the only state where wsrep_ready == "ON". | 10:32 |
| gaudenz_ | I'm not sure if wsrep_local_state == 2 should also be considered as a valid node status for the playbook to continue. The current code did not care about the node status at all, as the first condition was always false if any node status could be read. | 10:35 |
| gaudenz_ | As I understand it now from the proxysql docs, 4 means Synced and 2 means Donor. So it's probably OK to continue in these two states. In both cases the node should be healthy. | 10:37 |
| noonedeadpunk | yeah, as it switches to 2 while running mariabackup | 10:38 |
| noonedeadpunk | I guess my biggest concern about comment field - is how consistent it is. As most try to rely on ids rather then comments | 10:39 |
| gaudenz_ | My thought it's better to rely on a string field wich has a speaking value than an undocumented id. But I'll update the change to check for "not in ["2", "4"]". Now that I better understand it this seams the correct thing to do. | 10:40 |
| noonedeadpunk | ++ | 10:42 |
| opendevreview | Gaudenz Steinlin proposed openstack/openstack-ansible-galera_server master: Fix condition in cluster state checks https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922375 | 10:47 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Remove handler delegation for DB contract https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/921851 | 10:54 |
| jrosser | noonedeadpunk: https://paste.opendev.org/show/bewgkQDqXEdKIceWBwaJ/ | 11:11 |
| jrosser | thats from doing `openstack container list` in the utility container | 11:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2024.1: Remove handler delegation for DB contract https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/922387 | 11:28 |
| noonedeadpunk | well, it's still 401 from rgw? | 11:28 |
| jrosser | https://paste.opendev.org/show/bbhr8mRhzjntMwpaxxt8/ | 11:33 |
| noonedeadpunk | um. | 11:34 |
| jrosser | indeed | 11:34 |
| noonedeadpunk | why `radosgw`.... | 11:34 |
| jrosser | because https://codesearch.opendev.org/?q=radosgw_service_name&i=nope&literal=nope&files=&excludeFiles=&repos= | 11:36 |
| noonedeadpunk | so ceph-rgw-keystone-setup.yml doesn't execute? | 11:37 |
| jrosser | it does, i see it in my ansible.log | 11:38 |
| jrosser | and also the user and service exist (they're at the bottom of the paste) | 11:39 |
| jrosser | this is pretty confusing because it's still reef, and the only thing we switch out is ceph-ansible | 11:39 |
| jrosser | it's not like we should be finding some behaviour change in ceph itself | 11:40 |
| noonedeadpunk | so, if service is there... | 11:42 |
| noonedeadpunk | but keystone raises `Could not find service`... | 11:42 |
| noonedeadpunk | like - token is unscoped or having wrong permissions? | 11:42 |
| noonedeadpunk | are you checking zuul job? or reproduced in aio? | 11:43 |
| jrosser | i have an AIO that i made this morning | 11:43 |
| jrosser | and it reproduces what we see in zuul | 11:43 |
| jrosser | it makes little sense as there is a service named "radosgw" in `openstack service list` as well | 11:44 |
| noonedeadpunk | and can you list services if make an openrc file from ceph.conf creds? | 11:49 |
| jrosser | yes i seem to be able to | 11:55 |
| noonedeadpunk | but as you said - that's unlikely ceph on itself | 11:57 |
| noonedeadpunk | more how it's configured... | 11:58 |
| jrosser | the keystone integration is done in an OSA playbook so that should also not be changing | 11:59 |
| jrosser | maybe i just need to compare the ceph.conf with a stable-7.0 job and see if there is anything different | 12:00 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Update ssh_transfer_method parameter definition to match upstream https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/921837 | 13:37 |
| *** gaudenz_ is now known as gaudenz | 13:38 | |
| jrosser | i must be missing something really obvious with ceph rgw | 14:18 |
| mgariepy | can you paste your ceph.conf for rgw ? | 14:21 |
| jrosser | hmm well i did hack at it a bit | 14:22 |
| * jrosser re-runs rgw playbook | 14:22 | |
| mgariepy | hehe | 14:23 |
| jrosser | mgariepy: https://paste.opendev.org/show/bc3Aq6fisE6tK3sznEuC/ | 14:24 |
| mgariepy | isn't it a ssl redirect from keystone ? | 14:35 |
| mgariepy | anyone here have dealt with an undeletable bucket in radosgw ? | 14:37 |
| jrosser | mgariepy: where do you see the redirect? | 14:38 |
| jrosser | this is in an AIO with internal vip == http | 14:38 |
| mgariepy | when you auth to keystone in't the {"rel": "self", "href": ..} set to https ? | 14:38 |
| mgariepy | wget -q -O - http://172.29.236.101:5000 | 14:39 |
| noonedeadpunk | I can recall related patch to that | 14:41 |
| * jrosser not understanding how this relates to stable-7.0 -> stable-8.0 of ceph-ansible /o\ | 14:41 | |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/921312 | 14:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Set X-Forwarded-Proto header in apache conditionally https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/921312 | 14:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Remove legacy centos-8-stream job https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/922407 | 15:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove legacy centos-8-stream job definition https://review.opendev.org/c/openstack/openstack-ansible-tests/+/922408 | 15:46 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Update rabbitmq-server to 3.13.3 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/922378 | 15:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/922409 | 15:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts unmaintained/wallaby: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/922410 | 15:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible unmaintained/wallaby: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922412 | 15:57 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922413 | 16:00 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible unmaintained/yoga: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922414 | 16:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron unmaintained/wallaby: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/922415 | 16:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/922416 | 16:06 |
| NeilHanlon | jrosser: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/922409 and https://review.opendev.org/c/openstack/openstack-ansible/+/922413 looks like you just missed the second line on a couple of the definitions. i can propose a fix on them if you want | 16:12 |
| jrosser | ahha yes go for it | 16:12 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922413 | 16:16 |
| NeilHanlon | actually not sure what's up with 922409.. but.. might be transient | 16:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922413 | 16:20 |
| NeilHanlon | lol oh no.. what have we unearthed | 16:22 |
| jrosser | 8-O wtf is that :) | 16:23 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible unmaintained/xena: Remove EOL centos-8-stream jobs https://review.opendev.org/c/openstack/openstack-ansible/+/922413 | 16:31 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Update mariadb to 11.4.2 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922377 | 16:53 |
| *** jph5 is now known as jph | 17:46 | |
| opendevreview | Bjoern Teipel proposed openstack/openstack-ansible-os_keystone master: Remove X-Forwarded-Proto header in apache https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/921312 | 18:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!