| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 06:41 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 06:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Rename -noble nodeset to -ubuntu-noble https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 06:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Rename -noble nodeset to -ubuntu-noble https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 06:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename -noble nodeset to -ubuntu-noble https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 07:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename -noble nodeset to -ubuntu-noble and switch some jobs to 16GB https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 07:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Switch to wsgi module https://review.opendev.org/c/openstack/kolla-ansible/+/958070 | 07:27 |
| mnasiadka | let's see if octavia running on 16GB helps | 07:28 |
| mnasiadka | but it's funny, everything works until we run test-octavia.sh - then in the progress of it - ping timeouts to mariadb and everything is down | 07:28 |
| mnasiadka | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_91b/openstack/91b66a4311bf476187ee47d682ed9d38/primary/logs/kolla/proxysql/proxysql.txt | 07:29 |
| mnasiadka | ah | 07:33 |
| mnasiadka | 2025-08-21T20:24:51.430772+00:00 ubuntu kernel: CPU 0/TCG invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 | 07:33 |
| mnasiadka | that means it should help | 07:33 |
| mnasiadka | frickler: should we rename ubuntu node sets to have 8GB and 16GB suffix and switch jobs when they fail? | 07:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Check for OOM in check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/958265 | 07:40 |
| frickler | mnasiadka: sounds reasonable, can take a closer look at your changes later | 07:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Switch to wsgi module https://review.opendev.org/c/openstack/kolla-ansible/+/958070 | 07:52 |
| Vii | There's a chance you'll accept this? | 07:57 |
| Vii | Something happened with MariaDB, so a recheck is needed. Regarding the Vault tests, there are errors because a restart of the load balancer and HAProxy is required during deployment. They don't restart automatically after a service is added. There's also an error with Fluentd because parsers are missing, and we need to reconfigure common. These issues do not appear during a full deployment with Barbican. | 07:57 |
| Vii | https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 07:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename -noble nodeset to -ubuntu-noble and switch some jobs to 16GB https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 08:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename -noble nodeset to -ubuntu-noble and switch some jobs to 16GB https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 08:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename nodesets to reflect 8GB/16GB split https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 08:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename nodesets to reflect 8GB/16GB split https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 08:08 |
| Vii | mnasiadka: we can save some memory on the OpenStack services by reducing the number of workers. We can adjust the openstack_service_workers variable in the configuration file to start. | 08:08 |
| mnasiadka | yes, and it's already at "1" | 08:09 |
| Vii | this should probably help? | 08:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 08:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Switch to wsgi module https://review.opendev.org/c/openstack/kolla-ansible/+/958070 | 08:25 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 08:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Introduce service-container-bootstrap role https://review.opendev.org/c/openstack/kolla-ansible/+/958269 | 08:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Rename nodesets to reflect 8GB/16GB split https://review.opendev.org/c/openstack/kolla-ansible/+/958262 | 10:17 |
| Vii | A simple fix, please review/approve it for https://review.opendev.org/c/openstack/kolla-ansible/+/940810 | 10:39 |
| Vii | The monitoring system detects an additional backend that is offline. | 10:39 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 11:17 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 11:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 11:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Check for OOM in check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/958265 | 11:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Switch to wsgi module https://review.opendev.org/c/openstack/kolla-ansible/+/958070 | 11:44 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Introduce service-container-bootstrap role https://review.opendev.org/c/openstack/kolla-ansible/+/958269 | 11:47 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 11:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 11:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add support for Rocky Linux 10 https://review.opendev.org/c/openstack/kolla-ansible/+/956567 | 11:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add support for Rocky Linux 10 https://review.opendev.org/c/openstack/kolla-ansible/+/956567 | 11:58 |
| Lee | Greetings. I'm sure you get a lot of people making mistakes and then thinking its a Kolla bug. I hope I'm not one, but I fear I may be. I'm using Kayobe, which in turn is using kolla. It's failing due to ping working. I understand that the point of these separate networks is to segregate traffic, but what I don't understand is why the tool is expecting the ping to fail. I'm sure this isn't a bug with the ansible config or someone | 12:51 |
| priteau | Lee: Hello. Thanks for using Kayobe :) Can you share which task is failing exactly? | 12:53 |
| Lee | What I mean is, the tool is creating VLANs and subnets for me and assigning the IPs, such as 10.0.2.3/24, 10.0.3.3/24, 10.0.4.3/24, 10.0.5.3/24 and then it's running "ping 10.0.4.2" (ip of my kayobe/kolla machine) on each of my controller nodes. Why would it expect this ping to fail? If you run "ping 10.0.4.2" from a controller, it's going to send the ping on the eth0.4 (VLAN4) interface, which is on the same subnet and VLAN as 10 | 12:54 |
| Lee | If we want to test network segregation, shouldn't we be explicitly setting the interface such as "ping -I eth0.2 10.0.5.2"? I feel like I'm fundamentally misunderstanding what kayobe/kolla is trying to test or what it is attempting to configure. The specific TASK in question is "TASK [loadbalancer : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node]" | 12:54 |
| priteau | OK, that's purely a kolla-ansible task, not an issue with kayobe | 12:55 |
| Lee | Would greatly appreciate someone telling me where I'm just mis-interpreting/overlooking/otherwise being ignorant, but I'm really struggling. This is for a homelab, not business, but I'm far from a noobie when it comes to tech (worked as software engineer, devops, built out complex AWS infrastructures from scratch, and other stuff for 20 years). While I'm definitely not a networking expert, I'm definitely not a beginner either. | 12:57 |
| priteau | So, this task only runs when keepalived_vip_prechecks is true | 12:58 |
| priteau | This is a variable that is dynamically defined as: keepalived_vip_prechecks: "{{ all_hosts_in_batch and groups['keepalived_running_True'] is not defined }}" | 12:58 |
| priteau | So, it only runs if your keepalived container is not running | 12:59 |
| priteau | It is meant to check if the VIPs (both external and internal) are not accidently in use by another host | 12:59 |
| priteau | Are you sure you have configured your VIP addresses properly? They should not be in use if keepalived is not yet deployed | 13:00 |
| priteau | i.e. it is correct that ping should fail at this stage. | 13:00 |
| Lee | I see. I thought it was checking that the network isolation was working given that it's mentioning the external and internal kolla networks. That said, what is VIP in this context? This is to be like a floating IP? I've not explicitly configured anything along those lines. | 13:02 |
| Lee | Actually I found the config. I misinterpreted this. I thought it was intended to be the IPs of my machine running kayobe/kolla. The file is etc/kayobe/kolla.yml and the keys are kolla_internal_vip_address and kolla_external_vip_address, which I had set to my machine running kayobe. I've changed to an unused IP. | 13:05 |
| priteau | The VIP addresses are the IP used for the API endpoints when high-availability is enabled, which is by default. There is a bit of context here: https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/haproxy-guide.html | 13:07 |
| priteau | keepalived will pick a master controller node which will have the VIP assigned. If this node fails, it will be picked up by another controller. | 13:08 |
| Lee | OK yeah, I'm familiar with haproxy and heartbeat and such to automatically take over an IP when other hosts in the group fail. I had misinterpreted the description of the kolla_internal_vip_address and kolla_external_vip_address. | 13:09 |
| Lee | OK it's now passed this test after I changed to an unused IP rather than my kayobe machine IP. I greatly appreciate you pointing out my misunderstanding of what was happening / being tested. Thanks! | 13:16 |
| priteau | No problem | 13:41 |
| opendevreview | Seunghun Lee proposed openstack/kolla master: WIP: Bump mariadb to 11.8 https://review.opendev.org/c/openstack/kolla/+/957150 | 13:46 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible master: WIP: MariaDB 11.8 support https://review.opendev.org/c/openstack/kolla-ansible/+/958281 | 14:44 |
| Vii | mnasiadka: 2 weeks ago you did "fluentd: Split out to a separate role" and from what I see it doesn't work well, the service is not restarted after "reconfigure" | 15:03 |
| mnasiadka | And why would it? | 15:03 |
| Vii | PLAY [Apply role fluentd] ****************************************************** | 15:03 |
| Vii | PLAY [Apply role loadbalancer] ************************************************* | 15:03 |
| Vii | TASK [include_role : barbican] ************************************************* | 15:03 |
| Vii | look include and empty | 15:03 |
| mnasiadka | Don't paste Ansible logs here, use paste.openstack.org | 15:03 |
| Vii | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e60/openstack/e609ea4347054a2b82d9212592864cd8/primary/logs/ansible/reconfigure-barbican.log | 15:03 |
| Vii | Zulu: https://zuul.opendev.org/t/openstack/build/e609ea4347054a2b82d9212592864cd8/logs | 15:04 |
| mnasiadka | Vii: if you're running reconfigure --tags common,barbican - then fluentd is not going to run because https://opendev.org/openstack/kolla-ansible/src/commit/5170cc30755b90d18f2aeeb42d1ea583c5f020ea/ansible/site.yml#L113 | 15:07 |
| Vii | aaaaa Sorry, old habits | 15:07 |
| mnasiadka | Think before blaming other people :) | 15:07 |
| Vii | sorry | 15:07 |
| Vii | I'm testing the locale on code 2025.1 and only transfer the differences :/ | 15:09 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 15:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Check for OOM in check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/958265 | 15:11 |
| opendevreview | Doug Szumski proposed openstack/kolla master: Revert "Added systemd-standalone-sysusers package for Debuntu" https://review.opendev.org/c/openstack/kolla/+/958284 | 15:15 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Prevent accidental Libvirt downgrades https://review.opendev.org/c/openstack/kolla-ansible/+/942925 | 15:16 |
| opendevreview | Doug Szumski proposed openstack/kolla master: Allow fluentd to read the systemd journal https://review.opendev.org/c/openstack/kolla/+/955388 | 15:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 15:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 15:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add support for Rocky Linux 10 https://review.opendev.org/c/openstack/kolla-ansible/+/956567 | 15:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add support for Rocky Linux 10 https://review.opendev.org/c/openstack/kolla-ansible/+/956567 | 15:49 |
| opendevreview | Rafael Weingartner proposed openstack/kolla-ansible master: Add support to Dynamic ceilometer compute pollsters https://review.opendev.org/c/openstack/kolla-ansible/+/857501 | 16:32 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Adds selinux_update_kernel_param https://review.opendev.org/c/openstack/kayobe/+/957878 | 17:49 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "CI: Make rocky9 VM jobs non-voting" https://review.opendev.org/c/openstack/kayobe/+/958299 | 18:47 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.1: Revert "CI: Make rocky9 VM jobs non-voting" https://review.opendev.org/c/openstack/kayobe/+/958300 | 18:48 |
| opendevreview | Rafael Weingartner proposed openstack/kolla-ansible master: Add support to Dynamic ceilometer compute pollsters https://review.opendev.org/c/openstack/kolla-ansible/+/857501 | 19:10 |
| opendevreview | Rafael Weingartner proposed openstack/kolla-ansible master: Add support to Dynamic ceilometer compute pollsters https://review.opendev.org/c/openstack/kolla-ansible/+/857501 | 19:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 20:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 20:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 20:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Check for OOM in check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/958265 | 20:25 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!