| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/941315 | 07:17 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/941315 | 07:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/941316 | 07:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/941317 | 07:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/941318 | 07:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/941319 | 07:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_aodh master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/941320 | 07:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/941321 | 07:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/941322 | 07:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/941323 | 07:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/941325 | 07:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/941329 | 07:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/941330 | 07:57 |
| noonedeadpunk | NeilHanlon: btw, any news about glusterfs on EL10? | 07:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/941331 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/941332 | 08:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/941333 | 08:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/941334 | 08:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/941335 | 08:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/941336 | 08:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/941337 | 08:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/941338 | 08:12 |
| jrosser | we have an osa related question here https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/UWPZJ2V5U5A3HFW4LNMMHGSCB5RVLR7O/ | 08:50 |
| noonedeadpunk | I totally missed that | 09:09 |
| noonedeadpunk | but enabling extras by us makes total sense | 09:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/941342 | 09:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/941344 | 09:15 |
| noonedeadpunk | though I'd expect to be enabled from base images... | 09:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Respect defined neutron_dnsmasq_dns_servers for OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941351 | 10:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Simplify condition on ovn_ssl configuration in ml2 https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941353 | 10:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Respect defined neutron_dnsmasq_dns_servers for OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941351 | 10:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Simplify condition on ovn_ssl configuration in ml2 https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941353 | 10:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Simplify condition on ovn_ssl configuration in ml2 https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941353 | 10:22 |
| opendevreview | Jonathan Heathcote proposed openstack/openstack-ansible-os_ironic stable/2024.2: Migrate to from removed idrac-wsman to idrac-redfish https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/941354 | 10:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/941359 | 11:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/941360 | 11:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/941361 | 11:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/941364 | 11:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/941366 | 12:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/941367 | 12:28 |
| noonedeadpunk | we also finally need to look at adjutant and minalla..... | 12:29 |
| noonedeadpunk | *manilla | 12:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/941368 | 12:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/941369 | 12:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/941370 | 12:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/941371 | 12:36 |
| admin1 | hi .. if i need to upgrade from 25.2.0 to latest, what is the recommended way of doing it .. do i go to 26.0 and then 26.5, then 27.0 , 27.6 and then 28 ( first and last) or can i skip some in between ? | 12:43 |
| admin1 | just got an OK today for an upgrade :) | 12:43 |
| noonedeadpunk | 25.2 is Yoga, right? | 12:44 |
| admin1 | yeah | 12:44 |
| noonedeadpunk | you can upgrade to antelope, and then to caracal | 12:44 |
| noonedeadpunk | skipping zed and bobcat | 12:44 |
| noonedeadpunk | and then I'd suggest stopping on caracal and do just slurp releases, where you can jump in between :) | 12:44 |
| admin1 | so in terms of the tag system we use, 25 -> 27 -> 29 | 12:45 |
| noonedeadpunk | yeah | 12:46 |
| admin1 | do i need to use 27.0.0 and then to 29.0.0 or can also use the end ones 27.6.2 and 29.2.0 ? | 12:46 |
| noonedeadpunk | just do latest | 12:46 |
| admin1 | ok | 12:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941376 | 12:47 |
| admin1 | thanks noonedeadpunk .. on it | 12:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941378 | 12:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941376 | 12:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941378 | 12:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941376 | 12:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941378 | 12:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/941382 | 12:56 |
| harun | Hi all, I want to ask a question. I use Magnum and Vexxhost Cluster API Driver to create Cluster, but creating OpenStack instance takes too much time because of the slow storage driver. My question is that I want to increase the creation timeout of OpenStack instance and export CLUSTER_API_OPENSTACK_INSTANCE_CREATE_TIMEOUT variable but I don't know how to export and where. I tried exporting on k8s containers but I think it doesn't work. | 12:56 |
| noonedeadpunk | sooo... it should be a some kind of setting supported in the driver I think | 12:57 |
| harun | i looked but couldn't see if there was | 12:58 |
| noonedeadpunk | I think what you can try doing - is set `global_environment_variables` variable in /etc/openstack_deploy/group_vars/k8s_all.yml | 13:01 |
| noonedeadpunk | after running `openstack-ansible playbooks/lxc-container-create.yml --limit k8s_all,localhost --tags openstack_hosts-config` - env var should be added to /etc/environment | 13:01 |
| noonedeadpunk | But I have no idea if k8s cluster will just pick it up or not... | 13:02 |
| harun | Do you think nodeStartupTimeout in machineHealthCheck could work? | 13:03 |
| noonedeadpunk | I have really no idea. but machine spawn time more then 5 mins is kind of crazy | 13:04 |
| admin1 | so far i have been using tf to create the network and servers and do k8s manually .. | 13:04 |
| admin1 | what is the normal vm create/startup time if not using via magnum ? | 13:05 |
| admin1 | is it only slow via magnum or generally slow in your case ? | 13:05 |
| noonedeadpunk | I don't think capi should be adding any significancy to that | 13:06 |
| harun | we are trying to create instance via hitachi ... :/ | 13:07 |
| harun | booting from cinder | 13:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/941388 | 13:07 |
| noonedeadpunk | so Iassume you;ve adjusted nova timeout on it's own but then capi did introduce another level where timeout needs to be overriden :( | 13:08 |
| harun | the volume creation takes about 8 min | 13:08 |
| noonedeadpunk | /o\ | 13:08 |
| * noonedeadpunk writes down to stop complaining about storage performance | 13:09 | |
| admin1 | noonedeadpunk, what container/log do i need to tail/check to find out the failure for FAILED - RETRYING: [c1]: Ensure that the LXC cache has been prepared (113 retries left). -- this step ? | 13:11 |
| harun | thank you for your responses. i will try to solve this problem | 13:11 |
| harun | It is completely due to storage | 13:12 |
| noonedeadpunk | it's a valid retry | 13:12 |
| noonedeadpunk | if it goes to zero and does not suceed - then it's bad | 13:13 |
| admin1 | `fatal: [c1]: FAILED! => {"ansible_job_id": "782631210759.3693015", "attempts": 17, "changed": true, "cmd": "chroot /var/lib/machines/ubuntu-22-amd64 /opt/cache-prep-commands.sh > /var/log/lxc-cache-prep-commands.log 2>&1", "delta": "0:02:40.242266", "end": "2025-02-12 13:11:31.236169", "finished": 1, "msg": "non-zero return code", "rc": 100, | 13:13 |
| admin1 | "results_file": "/root/.ansible_async/782631210759.3693015", "start": "2025-02-12 13:08:50.993903", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}` | 13:13 |
| noonedeadpunk | then /var/log/lxc-cache-prep-commands.log :) | 13:13 |
| admin1 | oh i see | 13:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/941394 | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/941396 | 13:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/941397 | 13:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/941398 | 13:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/941400 | 13:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/941402 | 13:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/941403 | 13:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/941404 | 13:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/941316 | 13:35 |
| noonedeadpunk | roughly half of roles done... | 13:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/941410 | 13:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/941412 | 13:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/941418 | 14:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/941422 | 14:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/941425 | 14:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/941430 | 14:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/941433 | 14:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/941437 | 14:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/941438 | 14:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/941440 | 14:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/941444 | 14:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/941445 | 14:26 |
| admin1 | upgrade fails at TASK [openstack.osa.db_setup : Create database for service ,, does it have anything to do with ssl in the db ? | 14:28 |
| noonedeadpunk | um, depends. try re-running playbook with `-e _oslodb_setup_nolog=False` | 14:30 |
| noonedeadpunk | can have smth to do with absent /root/.my.cnf for old deployments | 14:36 |
| noonedeadpunk | (at times when root user was root and auth through password) | 14:36 |
| admin1 | "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: Packet sequence number wrong - got 1 expected 2 | 14:36 |
| admin1 | mysql is runinng though | 14:36 |
| noonedeadpunk | but can you login to it with mysql command from that host? | 14:37 |
| admin1 | mysql from util gives `mysql ERROR 2026 (HY000): TLS/SSL error: Broken pipe (32)` .. | 14:37 |
| admin1 | from the keystone container ? | 14:37 |
| admin1 | or in the db ? | 14:37 |
| admin1 | in galera, db is working fine | 14:37 |
| admin1 | from util, getting ssl error | 14:37 |
| noonedeadpunk | I don't see where your task failed? | 14:37 |
| noonedeadpunk | yeah, well | 14:37 |
| noonedeadpunk | I guess then it depends on your overrides... | 14:38 |
| noonedeadpunk | and content in my.cnf | 14:38 |
| noonedeadpunk | as, from some mariadb version tls encryption become pretty much required, but not 100% sure | 14:38 |
| noonedeadpunk | never tried to avoid it for mariadb/rabbitmq | 14:39 |
| admin1 | i see galera_ssl_verify: false , but nothing else in terms of not using ssl | 14:39 |
| noonedeadpunk | (though we should be having non-tls setup in ci) | 14:39 |
| noonedeadpunk | what's in my.cnf | 14:39 |
| admin1 | in util or in galera container ? | 14:39 |
| noonedeadpunk | if it fails from util - then in util | 14:40 |
| noonedeadpunk | I'm more interested in ssl-related options | 14:40 |
| admin1 | let me pastebin the whole vars and files | 14:40 |
| admin1 | noonedeadpunk https://gist.github.com/a1git/9f61cb32d4b2dd2abe846c0953b59b19 | 14:50 |
| noonedeadpunk | admin1: and what about output in `cat /etc/mysql/mariadb.cnf | grep ssl` from galera container? | 15:09 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic stable/2024.2: Migrate to from removed idrac-wsman to idrac-redfish https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/941354 | 15:33 |
| admin1 | noonedeadpunk, i see in the galera itself after ssl, the ssl-ca, cert and key paths, which is missing from the util | 15:40 |
| admin1 | i will copy those to util and redo the step to see if that fixes it | 15:40 |
| noonedeadpunk | yeah, they should not be needed... | 15:40 |
| admin1 | oh | 15:41 |
| noonedeadpunk | as CA should be in list of trusted, I guess | 15:41 |
| noonedeadpunk | you can try though | 15:41 |
| admin1 | shouldn't setup infra be fixing all these though ? | 15:41 |
| admin1 | which i ran twice already | 15:41 |
| noonedeadpunk | no, if some vars are not correct... | 15:41 |
| noonedeadpunk | and commenting out `ssl` from my.cnf does work? | 15:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Auto-fix usage of modules via FQCN https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/941364 | 15:44 |
| admin1 | mysql -h 172.29.236.9 -u admin -p -> Enter password: -> ERROR 2026 (HY000): TLS/SSL error: Broken pipe (32) | 15:44 |
| admin1 | somehow the server now has ssl enabled and clients(including util for running setup-osa ) cannot connect now | 15:45 |
| noonedeadpunk | what if add `--skip-ssl` to mysql command? | 15:45 |
| noonedeadpunk | but that somehow is weird... | 15:46 |
| admin1 | ERROR 1130 (HY000): Proxy header is not accepted from 172.29.236.9 | 15:46 |
| noonedeadpunk | oh, well | 15:46 |
| noonedeadpunk | I can recall smth like that | 15:46 |
| noonedeadpunk | 172.29.236.9 is balancer IP? | 15:46 |
| admin1 | yeah | 15:47 |
| admin1 | i already have galera_monitoring_allowed_source: 172.29.236.0/22 in variables .. | 15:47 |
| admin1 | proxy is new to me | 15:47 |
| noonedeadpunk | I bet we had written some reno about that | 15:47 |
| noonedeadpunk | as I think mariadb does indeed support proxy protocol only from some specific version | 15:49 |
| noonedeadpunk | but I'd expect it to be upgraded | 15:49 |
| admin1 | i will re-run setup hosts and infra without any flags and see if that updates anything | 15:51 |
| jrosser | you could check if the running version of mariadb is the one you expect | 15:51 |
| noonedeadpunk | I'd do that, yes | 15:51 |
| noonedeadpunk | as to upgrade mariadb you do need a flag for sure | 15:52 |
| admin1 | it says maria is 10.11.2 .. i have kind of never checked the verisons of rabbit or mysql .. just ran playbooks, ensured its running and that is about it | 15:53 |
| noonedeadpunk | I can totally recall proxy protocol issue if some step is missing back in the days... when we've switched haproxy to use it for mariadb | 15:53 |
| noonedeadpunk | um | 15:53 |
| noonedeadpunk | are you upgrading from Yoga directly to caracal? | 15:53 |
| admin1 | no .. 25 -> 27 as you recommended | 15:53 |
| admin1 | 25.2.0 -> 27.6.2 | 15:54 |
| noonedeadpunk | ah, ok,ok yes | 15:54 |
| admin1 | i will let the playbooks run and check again .. if necesary will delete the util container and let it build up again .. hopefully wtih correct files | 15:54 |
| noonedeadpunk | it's not the util container anyway | 15:55 |
| noonedeadpunk | I'd guess it's haproxy | 15:55 |
| noonedeadpunk | pretty much this: https://opendev.org/openstack/openstack-ansible/src/branch/stable/2023.1/inventory/group_vars/galera_all.yml#L70-L71 | 15:56 |
| noonedeadpunk | ah | 15:57 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/commit/d3bb90e0506b1812f06766fec874f93f80d4346e | 15:57 |
| noonedeadpunk | probably you'd need to replace `galera_monitoring_allowed_source` with `galera_server_proxy_protocol_networks` or smth like that... | 15:57 |
| noonedeadpunk | admin1: ^ | 15:58 |
| noonedeadpunk | or just extend it | 15:58 |
| noonedeadpunk | as it seems you still have messed up keepalived setup... | 15:59 |
| noonedeadpunk | it will propagate proxy-protocol-networks option to /etc/mysql/mariadb.cnf | 16:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Auto-fix yaml rules https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/941382 | 16:03 |
| WireLost | Hey everyone! I tried deploying OSA AIO on Ubuntu 24.04 with `SCENARIO=aio_lxc_ceph`, but it failed due to missing Ceph APT repos. Retried with `ceph_origin: distro` & `ceph_pkg_source: distro`, but it failed later. What's the status of OSA with Ceph on Noble? It works fine on 22.04. | 16:13 |
| admin1 | added that haproxy_backend_server_options , reran the haproxy playbook .. but still in thesame error .. | 16:14 |
| admin1 | galera_server_proxy_protocol_networks should be the same 172.29.236.0/22 ? | 16:14 |
| admin1 | i see . i think i get the gist of it | 16:16 |
| admin1 | retrying .. | 16:16 |
| jrosser | WireLost: the ceph project does not release packages for noble unfortuntatley https://download.ceph.com/debian-squid/dists/ | 16:18 |
| noonedeadpunk | WireLost: so yes, there're no repos for noble, but, I did deploy it fine recently and it works as expected | 16:18 |
| jrosser | noonedeadpunk: you mean you used the uca or ubuntu ceph packages? | 16:19 |
| WireLost | Ceph 19 Squid is on Noble by default, no need to use UCA. | 16:19 |
| WireLost | Even if you add Dalmatian UCA, Ceph 19 is still from Ubuntu's main repos. | 16:20 |
| WireLost | I'll try again! | 16:20 |
| jrosser | just FYI, the ceph integration in OSA is really a test fixture | 16:20 |
| admin1 | mariadb.cnf:proxy-protocol-networks = 172.29.236.11, localhost, ::1 -- i should expect to also see 172.29.236.0/22 right ? | 16:20 |
| WireLost | jrosser, ok | 16:20 |
| noonedeadpunk | for me it was enough to set `ceph_origin: distro` & `ceph_pkg_source: distro` & `ceph_mgr_modules: ['prometheus']` | 16:20 |
| jrosser | so for production deployments we always suggest a seperate ceph deployment | 16:21 |
| WireLost | noonedeadpunk, cool! I'll try it. thanks! | 16:21 |
| jrosser | but what noonedeadpunk pastes above will get you going | 16:21 |
| noonedeadpunk | as there was smth off with dashboard plugin and some python dependency | 16:21 |
| noonedeadpunk | admin1: so, I think your issue is, that you adding VIP with a wrong netmask | 16:22 |
| WireLost | jrosser, got it, but I do like how OSA is neatly integrated with Ceph Ansible, so that we can use `openstack-ansible` to deply Ceph with `ceph-ansible`, while leveraging OSA's inventory. | 16:22 |
| noonedeadpunk | which causes haproxy to go not with it's management IP, but with the VIP | 16:22 |
| admin1 | you meant haproxy_keepalived_internal_vip_cidr: "172.29.236.9/22" ? | 16:23 |
| noonedeadpunk | jrosser: btw we probably should add noble test for ceph after all... | 16:23 |
| noonedeadpunk | admin1: yes, it must be /32 | 16:23 |
| admin1 | changing, testing .. | 16:24 |
| noonedeadpunk | and then you don;t need to override galera_monitoring_allowed_source | 16:24 |
| noonedeadpunk | same applies for haproxy_keepalived_external_vip_cidr | 16:24 |
| noonedeadpunk | like if you check example: https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/user_variables.yml.prod.example#L8 | 16:25 |
| admin1 | did that, also added 2 vars galera_monitoring_allowed_source and galera_server_proxy_protocol_networks with "172.29.236.0/22" | 16:26 |
| noonedeadpunk | you should not need neither of them | 16:27 |
| admin1 | oh | 16:27 |
| noonedeadpunk | once keepalived configured correctly | 16:27 |
| admin1 | ok | 16:27 |
| jrosser | WireLost: the integration is nice, but you have 3 release cadences to deal with, your OS, openstack and ceph | 16:27 |
| noonedeadpunk | I can't resist but use integration as well for HCI 3-node clusters... | 16:28 |
| noonedeadpunk | I know it's bad... | 16:28 |
| jrosser | it just means that at some point there will be a total nightmare/impossible upgrade situation | 16:29 |
| noonedeadpunk | well. it's always like that with HCI? | 16:29 |
| noonedeadpunk | but in fact - I think it's all managable | 16:29 |
| noonedeadpunk | or well. | 16:29 |
| noonedeadpunk | unless you unlucky with ansible version on top | 16:30 |
| jrosser | well or a surprise, like we have had before with UCA | 16:30 |
| jrosser | well yes ansible version too as that has to be ok between OSA and ceph-ansible | 16:30 |
| WireLost | jrosser, good point! I see that the Ubuntu/UCA is being a bit weird after 24.04. For example, UCA for Noble is not documented on Ubuntu's wiki page | 16:31 |
| jrosser | so it all depends on context really, if it's a test lab or something you don't mind having to do some deep understanding on at upgrade time, then the integrated approach is ok | 16:32 |
| jrosser | if you want things to be more loosely coupled, so that ceph upgrades are inependant of openstack, then approach it differently | 16:33 |
| noonedeadpunk | well... if you pin ceph version explicitly... as well as ceph-ansible repo version... and is lucky enough... and know what you are doing... | 16:35 |
| noonedeadpunk | and ready to deal with all that... | 16:35 |
| noonedeadpunk | but indeed - it's intended mostly for testing | 16:37 |
| admin1 | worked now .. thanks noonedeadpunk jrosser .. running the rest of the playbooks | 16:39 |
| admin1 | any such changes you recall when doing 27 -> 29 after this upgrade has finsihed | 16:40 |
| noonedeadpunk | um, there's migration to quorum queues | 16:41 |
| noonedeadpunk | there's a link and a note in upgrade guide | 16:41 |
| admin1 | this one (luckily) is a single controller setup | 16:41 |
| noonedeadpunk | so better to explicitly override to skip using quortum queues at the beginning, and enroll to them afterwards | 16:42 |
| noonedeadpunk | ah | 16:42 |
| noonedeadpunk | still you might be explicit about that as I can't recall what the logic is | 16:47 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Revert "Re-enable uWSGI as default for Neutron" https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/940459 | 17:49 |
| admin1 | on 27, running openstack-ansible os-neutron-install.yml or similar for nova gives an unlimted Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml " | 18:28 |
| admin1 | output in the terminal ? | 18:28 |
| noonedeadpunk | um? | 18:42 |
| admin1 | https://gist.github.com/a1git/89cf88c852dbddab9bb79a0aa2ad1f94 | 18:43 |
| noonedeadpunk | that is very weird | 18:48 |
| noonedeadpunk | also I can't reproduce that in my AIO | 18:51 |
| noonedeadpunk | did you happen to upgrade ubuntu in between? | 18:52 |
| noonedeadpunk | on the deploy host? | 18:52 |
| admin1 | i did not | 19:01 |
| admin1 | its not upgraded | 19:02 |
| jrosser | or bash is changed to some other shell? | 19:02 |
| admin1 | still bash | 19:02 |
| noonedeadpunk | I guess I'd try to drop /opt/ansible-runtime and re-run ./scripts/bootstrap-ansible.sh | 19:02 |
| noonedeadpunk | or just re-login to the deploy host | 19:03 |
| noonedeadpunk | (to clean out current ENV) | 19:03 |
| jrosser | there are templated scripts in /usr/local/bin too | 19:03 |
| jrosser | I think? | 19:04 |
| admin1 | rm -rf /opt/ansible-runtime and re-running bootstrap | 19:04 |
| noonedeadpunk | not sure about templated... | 19:04 |
| admin1 | now ok :) | 19:06 |
| admin1 | openstack server list and network list giving issue .. checking | 19:06 |
| admin1 | i guess i need to revisit my vars .. Feb 12 19:07:23 c1-neutron-server-container-b59a8674 neutron-server[12798]: 2025-02-12 19:07:23.865 12798 ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: vxlan. Service terminated! | 19:07 |
| admin1 | dunno what this means .. the vars and other yml was not modified | 19:09 |
| noonedeadpunk | well, if you didn't execute upgrade script - you could get ovn... | 19:10 |
| noonedeadpunk | as default was flipped on antelope | 19:10 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/stable/2023.1/scripts/upgrade-utilities/define-neutron-plugin.yml | 19:11 |
| noonedeadpunk | so if it's lxb (prior default), you'd need to set `neutron_plugin_type: ml2.lxb` and `neutron_ml2_drivers_type: "flat,vlan,vxlan,local"` and `neutron_plugin_base: ['router', 'metering']` to preserve behaviour | 19:12 |
| noonedeadpunk | pretty much what playbook does | 19:12 |
| admin1 | should't this have alerted ? openstack-ansible "${SCRIPTS_PATH}/upgrade-utilities/deploy-config-changes.yml" | 19:27 |
| admin1 | ? | 19:27 |
| noonedeadpunk | um, no | 19:27 |
| noonedeadpunk | they inlcuded all separately | 19:28 |
| noonedeadpunk | and we have mentioned that one in upgrade docs: https://docs.openstack.org/openstack-ansible/2023.1/admin/upgrades/major-upgrades.html#implement-changes-to-osa-configuration | 19:28 |
| NeilHanlon | noonedeadpunk: not directly about glusterfs, but we had a SIG/Storage kick-off meeting today for Rocky and folks are going to be working on packaging ceph. so, not really an answer except to say it's still on my radar 😅 | 19:29 |
| noonedeadpunk | so pretty much worth looking for replacement already? | 19:30 |
| noonedeadpunk | doh, would be tight together with 10 | 19:31 |
| admin1 | my bad .. i somehow missed the step :* | 19:33 |
| NeilHanlon | I think we can probably make it happen, I just need to ... do it :D | 19:40 |
| admin1 | tomorrow will do 27 -> 29 | 19:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2024.2: Revert "Re-enable uWSGI as default for Neutron" https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/941484 | 20:36 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!