| opendevreview | Michal Arbet proposed openstack/kolla master: Prepare /var/log/kolla in the base start script https://review.opendev.org/c/openstack/kolla/+/985300 | 00:05 |
|---|---|---|
| opendevreview | Michal Arbet proposed openstack/kolla master: fluentd: add remote_syslog output plugin support https://review.opendev.org/c/openstack/kolla/+/984472 | 00:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fix ovn-upgrade jobs https://review.opendev.org/c/openstack/kolla-ansible/+/985346 | 05:19 |
| mikal | Hey mnasiadka and friends, I'm just confirming the me-friendly session is in 24 hours, and not 15 minutes? That is, I am hoping I can still do timezone maths... | 06:43 |
| mnasiadka | mikal: yes, 24 hours :) | 06:43 |
| mikal | Sweet. Thank you. | 06:43 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: logging: unify Apache and uWSGI logs https://review.opendev.org/c/openstack/kolla-ansible/+/985219 | 07:48 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: fluentd: improve multiline firstline detection for OpenStack logs https://review.opendev.org/c/openstack/kolla-ansible/+/984463 | 07:48 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Refactor format1 to improve OpenStack context parsing https://review.opendev.org/c/openstack/kolla-ansible/+/984464 | 07:48 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: fluentd: replace Socket.gethostname https://review.opendev.org/c/openstack/kolla-ansible/+/984465 | 07:48 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: fluentd: parse HAProxy syslog messages with syslog parser https://review.opendev.org/c/openstack/kolla-ansible/+/984200 | 07:48 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add rsyslog https://review.opendev.org/c/openstack/kolla-ansible/+/985220 | 07:48 |
| opendevreview | Maximilian Stinsky proposed openstack/kolla master: Add strongswan plugins for VPNaaS https://review.opendev.org/c/openstack/kolla/+/985628 | 08:01 |
| tafkamax | Created an launchpad bug for sysctl | 08:07 |
| tafkamax | https://bugs.launchpad.net/kolla-ansible/+bug/2149783 | 08:08 |
| tafkamax | So this gets documented and we can think on it. | 08:08 |
| tafkamax | Well todays upgrade went smooooth | 08:08 |
| tafkamax | as we got this issued fixed ^ | 08:08 |
| tafkamax | chemberv_: We got our potential mariadb backup fix implemented. Currently it has been doing it for 1.5 hours | 08:40 |
| tafkamax | Our log is at this stage... (full message at <https://matrix.org/oftc/media/v1/media/download/AZzpFBeIbZ81eKhUcyIEEoGb3m3bBgJzAUGcu5KS8BgCaYzMLXZz52EQhmzKZzI8ltavP9XcILDZLD-9SOzjZztCed90x7BAAG1hdHJpeC5vcmcvSGZackVMSnpEaFF1clBqaXJuSnplT0FH>) | 08:41 |
| tafkamax | been like this for quite a long time. | 08:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache https://review.opendev.org/c/openstack/kolla-ansible/+/982414 | 08:47 |
| tafkamax | chemberv_: well looking at https://dba.stackexchange.com/questions/342017/unable-to-create-backup-with-mariabackup-try-increasing-innodb-log-file-size, the log file size thing might not be the whole story unfortunately | 08:51 |
| opendevreview | Owen Jones proposed openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 08:59 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Remove reference to deprecated app.wsgi in Aodh image https://review.opendev.org/c/openstack/kolla/+/985640 | 10:44 |
| tafkamax | mariadb backup failed finally | 11:01 |
| tafkamax | `[00] 2026-04-21 13:03:11 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:12 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:13 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:14 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:15 Was only able to copy log from 79628484036 to 79666092661, not 80404596117; try increasing innodb_log_file_size", "mariabackup: Stopping log | 11:01 |
| tafkamax | copying thread.[00] 2026-04-21 13:03:15 Retrying read of log at LSN=79666092661", ""],` | 11:01 |
| tafkamax | took a lot of time | 11:01 |
| tafkamax | I think it just cant keep up with the new amount of entries | 11:02 |
| tafkamax | > Each record in the log needs to be able to be redone multiple times since at restart and media recovery we will not know if that action has previously been executed as we will start recovery from the last checkpoint.InnoDB achieves this by reading the LSN of the page and compares that to the one of log record, if the LSN is higher that means the page was flushed after this record was processed so the change does not need to be | 11:02 |
| tafkamax | applied. | 11:02 |
| tafkamax | for mariadb before 11 might need to try this: https://mariadb.com/docs/server/server-usage/storage-engines/innodb/innodb-system-variables#innodb_flush_method | 11:02 |
| tafkamax | for faster flushing? | 11:03 |
| opendevreview | Owen Jones proposed openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 11:06 |
| tafkamax | I see that in the kolla repo the parrallel option is not used: https://opendev.org/openstack/kolla/src/branch/master/docker/mariadb-server/backup_replica.sh#L31 | 11:19 |
| tafkamax | Parallel docs: https://mariadb.com/docs/server/server-usage/backup-and-restore/mariadb-backup/mariadb-backup-options#parallel | 11:19 |
| tafkamax | Hmm this is an interesting one: https://review.opendev.org/c/openstack/kolla/+/978300 | 11:23 |
| kevko | tafkamax: what is the problem with mariadb backup ? | 12:15 |
| tafkamax | kevko: It fails with: Retrying read of log at LSN=79493095600", "[00] 2026-04-20 20:56:29 Retrying read of log at LSN=79493095600", "[00] 2026-04-20 20:56:30 Was only able to copy log from 78599448186 to 79493095600, not 81577467335; try increasing innodb_log_file_size", "maria | 12:16 |
| tafkamax | backup: Stopping log copying thread.[00] 2026-04-20 20:56:30 Retrying read of log at LSN=79493095600", ""], "stdout": "Taking a full backup\n", "stdout_lines": ["Taking a full backup"]} | 12:16 |
| tafkamax | We increased the innodb_log_file_size from default to 16GB and it failed after 3+ hours with the same error. | 12:17 |
| tafkamax | [00] 2026-04-21 13:03:11 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:12 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:13 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:14 Retrying read of log at LSN=79666092661", "[00] 2026-04-21 13:03:15 Was only able to copy log from 79628484036 to 79666092661, not 80404596117; try increasing innodb_log_file_size", "mariabackup: Stopping log | 12:17 |
| tafkamax | copying thread.[00] 2026-04-21 13:03:15 Retrying read of log at LSN=79666092661", ""], | 12:17 |
| tafkamax | I am starting to believe the error is with too many writes happening, so the backup can't keep up with new entries and it is an loop until the log file is maxxed out. | 12:18 |
| tafkamax | Adding 16gb makes it run longer as it has more storage to try again | 12:18 |
| kevko | tafkamax: is it backuping replica or master ? | 12:25 |
| kevko | tafkamax: because this i've already seen some time ago and I was actually the author of patch which switched to replica | 12:25 |
| kevko | tafkamax: because of ^^ | 12:26 |
| tafkamax | I don't know actually | 12:26 |
| tafkamax | How can I check it easily? | 12:26 |
| tafkamax | So it should run on the replica node? | 12:26 |
| tafkamax | Or on the master node? | 12:27 |
| kevko | tafkamax: https://review.opendev.org/c/openstack/kolla/+/928956 | 12:27 |
| tafkamax | we have that included yes | 12:28 |
| kevko | tafkamax: well, ussually you want to backup replica server right ? kolla-ansible using proxysql as default ...you can try to set replica to OFFLINE ..so proxysql will not send anything to that host ..and then you can retry ( but you need to do it manually) | 12:29 |
| tafkamax | For kolla-ansible I did cherry pick this change to test it out: https://review.opendev.org/c/openstack/kolla-ansible/+/978299?usp=search | 12:31 |
| tafkamax | Also what do you think of this patch? https://review.opendev.org/c/openstack/kolla/+/978300 | 12:31 |
| tafkamax | kevko: Oh okay, I don't know that much about proxysql and so on. | 12:31 |
| jovial[m] | Running out of disk space on the kayobe upgrade jobs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6ed/openstack/6edd10b99d8a4aecab817a28930de7e2/primary/system_logs/df.txt. Anyone know if one of the other nodesets provides a larger disk? | 12:34 |
| kevko | tafkamax: can be, let me check | 12:39 |
| jovial[m] | is it a bit random? You can land on a provider that only gives 40G, whilst another provider will give you 80G for the same nodeset (ubuntu-noble-16GB) | 12:43 |
| jovial[m] | https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/providers.yaml#L72-L82 vs https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/providers.yaml#L401-L403 | 12:44 |
| opendevreview | Bartosz Bezak proposed openstack/kolla master: [DNM] test bifrost on arm https://review.opendev.org/c/openstack/kolla/+/957672 | 12:54 |
| opendevreview | Bartosz Bezak proposed openstack/kolla master: [DNM] test bifrost on arm https://review.opendev.org/c/openstack/kolla/+/957672 | 12:56 |
| opendevreview | Owen Jones proposed openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 13:00 |
| opendevreview | Owen Jones proposed openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 13:00 |
| opendevreview | Owen Jones proposed openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 13:00 |
| blanson[m] | oh the ptg slot has been moved | 13:10 |
| blanson[m] | as you can see I am very organized | 13:11 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Support configuring DNS on Ironic Neutron networks https://review.opendev.org/c/openstack/kayobe/+/971027 | 13:21 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Adds option to disable registration of networks https://review.opendev.org/c/openstack/kayobe/+/971121 | 13:21 |
| opendevreview | Bartosz Bezak proposed openstack/kolla master: CI: sync Debian scenario jobs with kolla-ansible https://review.opendev.org/c/openstack/kolla/+/985655 | 13:29 |
| opendevreview | Merged openstack/kayobe master: Add infra-vms to container-engine group https://review.opendev.org/c/openstack/kayobe/+/983712 | 15:08 |
| opendevreview | Merged openstack/kayobe master: Support provisioning hosts without kolla services https://review.opendev.org/c/openstack/kayobe/+/976525 | 15:08 |
| opendevreview | Merged openstack/kayobe master: Block until timedatectl status show synchronized https://review.opendev.org/c/openstack/kayobe/+/982396 | 15:08 |
| jovial[m] | mnasiadka, Do you do anything with the secondary drive in the kolla jobs? Just wondering if there is a pattern I can copy. | 15:15 |
| jovial[m] | https://docs.opendev.org/opendev/infra-manual/latest/testing.html#unprivileged-single-use-vms | 15:16 |
| mnasiadka | jovial[m]: kolla uses ephemeral drive - but it’s not available on all providers | 15:25 |
| kevko | anybody seen this in current CI jobs ? https://ba8cdf5843f45d47b0c9-7705306510805cbea58d1e45781b1e11.ssl.cf5.rackcdn.com/openstack/a66bef814edb4a149c17da8dce59187c/primary/logs/kolla/all-ERROR.txt | 15:30 |
| opendevreview | Will Szumski proposed openstack/kayobe master: DNM: Testing multiple block devices https://review.opendev.org/c/openstack/kayobe/+/985696 | 15:32 |
| mnasiadka | blanson[m]: you agreed to cancel the Tue slot :D | 15:40 |
| jovial[m] | mnasiadka, don't suppose you can link me to where you configure it? | 15:41 |
| mnasiadka | jovial[m]: https://opendev.org/openstack/kolla/src/commit/a9bae9fe56dced112e71de51bd9f382f00d584ff/tests/playbooks/pre.yml#L7 and https://opendev.org/openstack/kolla/src/branch/master/roles/configure-ephemeral | 15:41 |
| jovial[m] | lovely, thanks | 15:42 |
| mnasiadka | You can use the same role if that fits your purpose, or there should be another one under https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap | 15:42 |
| mnasiadka | jovial[m]: basically providers give you either one big disk or two small ones | 15:43 |
| opendevreview | Will Szumski proposed openstack/kayobe master: DNM: Testing multiple block devices https://review.opendev.org/c/openstack/kayobe/+/985696 | 15:46 |
| opendevreview | Will Szumski proposed openstack/kayobe master: DNM: Testing multiple block devices https://review.opendev.org/c/openstack/kayobe/+/985696 | 15:47 |
| jovial[m] | We seemed to be getting away with for a while but now are running out of disk space on the upgrade jobs (when just using the root disk), so that code will be very handy :) | 15:49 |
| mnasiadka | We usually use that for Docker/Podman | 15:52 |
| opendevreview | Merged openstack/kayobe stable/2025.2: Bump stackhpc.openstack to 0.10.1 https://review.opendev.org/c/openstack/kayobe/+/985401 | 16:40 |
| opendevreview | Merged openstack/kayobe stable/2025.2: Override IPA image distribution/release https://review.opendev.org/c/openstack/kayobe/+/985340 | 16:43 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: Fix ovn-upgrade jobs https://review.opendev.org/c/openstack/kolla-ansible/+/985346 | 16:45 |
| opendevreview | Will Szumski proposed openstack/kayobe master: DNM: Testing multiple block devices https://review.opendev.org/c/openstack/kayobe/+/985696 | 16:48 |
| opendevreview | Merged openstack/kayobe master: Add mechanism to add custom CAs to trust store https://review.opendev.org/c/openstack/kayobe/+/981811 | 17:19 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Support configuring DNS on Ironic Neutron networks https://review.opendev.org/c/openstack/kayobe/+/971027 | 17:24 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix issue with systemd-networkd-wait-online when no IP https://review.opendev.org/c/openstack/kayobe/+/975386 | 17:39 |
| -opendevstatus- NOTICE: The wiki.openstack.org service will be offline for a few minutes while we perform a database upgrade | 18:07 | |
| blanson[m] | mnasiadka: I did, I just have a goldfish's memory and too much work :D | 19:06 |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: Tidy up release notes for Gazpacho release https://review.opendev.org/c/openstack/kolla-ansible/+/983170 | 19:07 |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: keystone: Setup fernet credentials encryption keys https://review.opendev.org/c/openstack/kolla-ansible/+/970861 | 19:25 |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: keystone: Improve fernet/credential key bootstrap and distribution https://review.opendev.org/c/openstack/kolla-ansible/+/982113 | 19:25 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Fix ovn-upgrade jobs https://review.opendev.org/c/openstack/kolla-ansible/+/985346 | 20:37 |
| opendevreview | Merged openstack/kayobe master: Support configuring DNS on Ironic Neutron networks https://review.opendev.org/c/openstack/kayobe/+/971027 | 21:02 |
| *** jhorstmann is now known as Guest7713 | 22:20 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!