| *** noonedeadpunk_ is now known as noonedeadpunk | 08:14 | |
| noonedeadpunk | TheCompWiz: for clean/new deployment feel free to to re-run with `-e galera_ignore_cluster_state=true -e galera_force_bootstrap=true` as suggested by the role | 08:18 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: DNM Allow to skip pre-step bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/905290 | 09:22 |
| noonedeadpunk | ugh, spent couple of hours on trying to understand how dynamic_inventory checks for existing cotnainers and where it decides to create a new one - giving up.... | 10:26 |
| noonedeadpunk | I'ts jsut /o\ | 10:27 |
| noonedeadpunk | I thought it would be quite easy hack in dictutils, but no, decision to generate new UUID comes from somewhere else.... | 10:27 |
| noonedeadpunk | I guess I should have whined earlier to find the thing I'm looking for lol | 10:32 |
| admin1 | TheCompWiz, why are your container interfaces modified ? | 11:23 |
| Tadios | jrosser: the comment here https://github.com/openstack/openstack-ansible-ops/blob/1c0bd2ae111b57ba9366587d1c4542655b215d14/elk_metrics_6x/README.rst?plain=1#L54C21-L54C21 states that kibana nodes are also elasticsearch coordination nodes, and haproxy_backend_nodes for elastic-logstash is set to groups['Kibana'], is this accurate? | 11:29 |
| Tadios | or should the var haproxy_backend_nodes for elastic-logstash set to groups['elastic'], i'm confused | 11:29 |
| admin1 | noonedeadpunk, https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upgrades.html => "This release is under development. The current supported release is 2023.2." -- what does 28.0.0 tag correspond to , and is it SLURP release ? | 11:36 |
| admin1 | "This guide provides information about the upgrade process from 2023.1 to 2023.2 for OpenStack-Ansible." -> maybe we can also have corresponding tags ? | 11:37 |
| noonedeadpunk | admin1: tags are on the main: https://docs.openstack.org/openstack-ansible/latest/ | 11:41 |
| noonedeadpunk | 2023.2 is not SLURP and it's 28.0.0 | 11:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Treat dashes/underscores as interchangeable symbols for container names https://review.opendev.org/c/openstack/openstack-ansible/+/905432 | 11:48 |
| admin1 | aha .. so easy .. odd is slurp and even is non-slurp in our case as well | 11:55 |
| admin1 | actually not :D | 11:56 |
| admin1 | upgrading a platform from 27.0.1 -> 28.0.0 | 12:18 |
| noonedeadpunk | 27 is slurp btw | 12:30 |
| noonedeadpunk | admin1: fwiw our versioning is actually the same as for nova | 12:31 |
| noonedeadpunk | also all release information can be found here: https://releases.openstack.org/ | 12:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not use underscores in container names https://review.opendev.org/c/openstack/openstack-ansible/+/905433 | 12:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not use underscores in container names https://review.opendev.org/c/openstack/openstack-ansible/+/905433 | 13:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Allow env.d to contain underscores in physical_skel https://review.opendev.org/c/openstack/openstack-ansible/+/905438 | 13:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Allow env.d to contain underscores in physical_skel https://review.opendev.org/c/openstack/openstack-ansible/+/905438 | 13:32 |
| noonedeadpunk | I _think_ now we should be able to rename our env_d and get rid of Ansible warning regarding using dashes in group names | 13:32 |
| mgariepy | i'm a bit confused by that. i somewhat recall something about glance and uwsgi/haproxy and ceph ? https://bugs.launchpad.net/glance/+bug/1916482 | 13:33 |
| noonedeadpunk | Yeah, and I think there was fix for that proposed even | 13:36 |
| mgariepy | this ?https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/879699 | 13:38 |
| noonedeadpunk | mgariepy: I think this: https://review.opendev.org/c/openstack/glance_store/+/885581 | 13:54 |
| noonedeadpunk | or well. not using uwsgi works for us as well | 13:55 |
| mgariepy | i really don't like using code that isn't merged.. | 14:10 |
| mgariepy | even more when the patch has been sitting for months. | 14:12 |
| noonedeadpunk | yeah | 14:14 |
| mgariepy | not sure what has changed but the snapshot was workiong last october. and is broken now. that's really fun. | 15:05 |
| admin1 | i am upgrading one env to 28.0.0 .. how to validate/check if this patch -> https://review.opendev.org/c/openstack/neutron-fwaas/+/845756 is there, or how to use this kine of custom patch when using osa | 15:14 |
| noonedeadpunk | admin1: well, I'd wait for a next point release | 15:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic stable/2023.2: Stop generating ssh keypair for ironic user https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/903543 | 15:35 |
| noonedeadpunk | damiandabrowski: would be sweet if you could take a look at these backports: https://review.opendev.org/c/openstack/openstack-ansible/+/905321 https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/905081 | 15:36 |
| noonedeadpunk | So I could propose bumps for 2023.2 | 15:36 |
| NeilHanlon | i can review if you need | 15:41 |
| noonedeadpunk | yes, thanks, that will work as well :) | 15:45 |
| dmsimard[m] | btw if you'll be at upcoming FOSDEM: https://www.meetup.com/brussels-openinfra-meetup-group/events/298420649/ | 16:00 |
| noonedeadpunk | dmsimard[m]: thanks! | 16:05 |
| noonedeadpunk | Ugh, maybe it's worth going there after all.... | 16:05 |
| * noonedeadpunk got paperwork sorted | 16:05 | |
| NeilHanlon | thanks dmsimard[m]! | 16:42 |
| * NeilHanlon wonders if we just bullied noonedeadpunk into coming to FOSDEM... | 16:42 | |
| noonedeadpunk | need to check if anything interesting in the schedule :D | 16:45 |
| noonedeadpunk | (except beer) | 16:45 |
| NeilHanlon | heh | 16:49 |
| dmsimard[m] | ¯\_(ツ)_/¯ | 16:52 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/2023.2: Fix resolved config on Debian https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/905081 | 18:18 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Return back /healtcheck URI verification https://review.opendev.org/c/openstack/openstack-ansible/+/905321 | 18:41 |
| TheCompWiz | Is there any way to get ansible to re-deploy the database on a galera container? ... I'm not sure why, but after setup-infrastructure.yml, the database has none of the permissions set, and it appears as if no data has been set either. | 19:58 |
| TheCompWiz | https://paste.openstack.org/show/bo2OQsf0Han0dHhfC7EX/ | 20:03 |
| jrosser | TheCompWiz: the database will only have root and admin users after setup-infrastructure | 21:29 |
| jrosser | each service that gets deployed during setup-openstack creates the needed db and user at the point they are required | 21:30 |
| TheCompWiz | jrosser: ok... then any ideas why keystone fails on the task "Create database for service"? | 21:31 |
| jrosser | well, a combination of seeing the output of the task with no_log disabled would help | 21:31 |
| TheCompWiz | see above :) | 21:32 |
| jrosser | does the MySQL cli client work on the utility container? | 21:33 |
| TheCompWiz | ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 11 | 21:35 |
| TheCompWiz | hmmmm. | 21:35 |
| jrosser | ok, so that cli connects to the db via haproxy | 21:35 |
| jrosser | haproxy needs to think that the db backend is up | 21:35 |
| jrosser | you can check that using hatop on the haproxy node | 21:36 |
| TheCompWiz | hmm... showing all of the "galera" containers as down... but none are. | 21:44 |
| TheCompWiz | I can lxc-attach to each of them... and just do "mysql" and query everything. | 21:45 |
| TheCompWiz | (using root@localhost with no password) | 21:46 |
| jrosser | there is a an http healthcheck run on port 9200 which is what haproxy uses to determine the backend health | 21:55 |
| jrosser | from the perspective of haproxy that healthcheck much be failing if the backends are marked as down | 21:55 |
| TheCompWiz | I can connect to the mysql db directly from the ha node using the credentials that haproxy is supposedly using... | 22:04 |
| TheCompWiz | why is it using an http healthcheck? | 22:04 |
| jrosser | because haproxy does not understand port 3306 to determine the galera cluster status | 22:08 |
| jrosser | haproxy does not have credentials for the database | 22:08 |
| jrosser | a service is run on the galera nodes on port 9200 which returns good/bad depending on if each backend database node is successfully part of the db cluster | 22:09 |
| jrosser | it would be wrong to mark a db backend as good which was “working” but failed to be in the cluster | 22:10 |
| jrosser | so the http based healthcheck permits a more advanced status to be used by haproxy when deciding of the db backends are usuable or not | 22:11 |
| jrosser | try using curl on port 9200 to the ip of the galera container from the haproxy node | 22:11 |
| TheCompWiz | hmmm... I think I figured it out. The stupid mariadb check is only accepting connections from the "external_lb_vip_address" and not the "internal_lb_vip_address" | 22:17 |
| TheCompWiz | ... or in-short... from my haproxy server... it has 2 IPs it *could* use... and it picked the wrong one. | 22:19 |
| TheCompWiz | ... /grumble. | 22:19 |
| jrosser | TheCompWiz: it is defined here https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/galera_all.yml#L33-L39 | 22:21 |
| jrosser | this is happening because you have specified 192.168.122.100 as the IP for your loadbalancer node in haproxy_hosts | 22:23 |
| jrosser | is it not correct to use either of the VIP addresses as the IP for that node in openstack_user_config | 22:24 |
| jrosser | in an H/A deployment the internal and external vip will float between all haproxy nodes using keepalived | 22:25 |
| jrosser | you need a unique and fixed address on the mgmt network for each node in openstack_user_config which then becomes `management_address` | 22:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!