| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Switch setup_gate.sh to use Kolla roles https://review.opendev.org/c/openstack/kolla-ansible/+/957136 | 06:45 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Test fwaas & vpnaas https://review.opendev.org/c/openstack/kolla-ansible/+/957205 | 06:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Test fwaas & vpnaas https://review.opendev.org/c/openstack/kolla-ansible/+/957205 | 06:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: Add support for deploying registry in kolla-build role https://review.opendev.org/c/openstack/kolla/+/957135 | 07:06 |
| opendevreview | Roman Krcek proposed openstack/kolla-ansible master: Move tasks from k-a role common to a-c-k https://review.opendev.org/c/openstack/kolla-ansible/+/948526 | 07:12 |
| opendevreview | Roman Krcek proposed openstack/kolla-ansible master: [POC] Rework configuration process https://review.opendev.org/c/openstack/kolla-ansible/+/946527 | 07:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Switch setup_gate.sh to use Kolla roles https://review.opendev.org/c/openstack/kolla-ansible/+/957136 | 07:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Test fwaas & vpnaas https://review.opendev.org/c/openstack/kolla-ansible/+/957205 | 07:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: nova: Fix extract_cells https://review.opendev.org/c/openstack/kolla-ansible/+/957210 | 08:15 |
| opendevreview | Mariusz Karpiarz proposed openstack/kolla stable/2025.1: Revert "mariadb: pin to 10.11.11" https://review.opendev.org/c/openstack/kolla/+/956538 | 08:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: pin ansible-core to <2.18.8 https://review.opendev.org/c/openstack/kolla-ansible/+/957211 | 08:51 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2025.1 failed: security: hide sensitive auth_password in kolla_container module logs https://review.opendev.org/c/openstack/kolla-ansible/+/957156 | 08:52 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Fix etcd backend certificates not being templated https://review.opendev.org/c/openstack/kolla-ansible/+/957217 | 10:46 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible master: Set default external Let's Encrypt cert server https://review.opendev.org/c/openstack/kolla-ansible/+/957141 | 11:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: neutron: Switch to uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/956785 | 11:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: pin ansible-core to <2.18.8 https://review.opendev.org/c/openstack/kolla-ansible/+/957211 | 11:52 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible master: Set default external Let's Encrypt cert server https://review.opendev.org/c/openstack/kolla-ansible/+/957141 | 12:14 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Prevent accidental downgrades of Libvirt https://review.opendev.org/c/openstack/kolla-ansible/+/942925 | 12:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Revert "security: hide sensitive auth_password in kolla_container module logs" https://review.opendev.org/c/openstack/kolla-ansible/+/957226 | 12:27 |
| opendevreview | Seunghun Lee proposed openstack/kolla master: WIP: Bump mariadb to 11.8 https://review.opendev.org/c/openstack/kolla/+/957150 | 12:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 12:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 12:49 |
| mnasiadka | bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew amir58118 r-krcek - meeting in 10 minutes | 12:50 |
| opendevreview | Amir Hossein Ahmadi proposed openstack/kolla-ansible master: swift: reintroduce Swift role and restore deployment https://review.opendev.org/c/openstack/kolla-ansible/+/953576 | 12:51 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Aug 13 13:00:54 2025 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| bbezak | o/ | 13:01 |
| amir58118 | \o | 13:01 |
| frickler | o/ | 13:01 |
| mnasiadka | #topic agenda | 13:04 |
| mnasiadka | * CI status | 13:04 |
| mnasiadka | * Release tasks | 13:04 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:04 |
| mnasiadka | * Current cycle planning | 13:04 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:04 |
| mnasiadka | * Open discussion | 13:04 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | It seems the sec bug merge broke our CI, common_opts no_log=True causes getting **** in some output from kolla_container - currently there's a patch up for revert, checking if it comes out green and fast-merging it | 13:05 |
| mnasiadka | Otherwise it was good yesterday I guess, so no other CI issues | 13:06 |
| mnasiadka | (other than some rax.ord mirror timeouts) | 13:06 |
| mnasiadka | which I already reported on #opendev | 13:06 |
| frickler | that's docker.io timeouts very likely | 13:06 |
| mnasiadka | Oh ok, maybe we should think about mirroring docker rpms/debs in opendev? | 13:07 |
| mnasiadka | (but that doesn't show up that often I guess) | 13:07 |
| opendevreview | Merged openstack/kolla stable/2024.1: Revert "mariadb: pin to 10.11.13" https://review.opendev.org/c/openstack/kolla/+/957182 | 13:08 |
| fungi | basically nothing on those per-provider "mirrors" is actual copies of anything aside from distro packages. all the other trees are caching proxies to external services like dockerhub or pypi | 13:08 |
| fungi | so if dockerhub delays or blocks responses to the proxy, then you'll get an error client-side | 13:09 |
| mnasiadka | In https://zuul.opendev.org/t/openstack/build/94be78a6d6e249cca2fb160f5eef3bac/log/primary/logs/ansible/bootstrap-servers#764 - it was during downloading gpg and docker client packages | 13:10 |
| mnasiadka | so not really dockerhub | 13:11 |
| mnasiadka | but I get the point | 13:11 |
| mnasiadka | let's move on | 13:12 |
| mnasiadka | #topic Release tasks | 13:12 |
| fungi | ah, if it was rpms or debs then that might be a connectivity problem between the client and the mirror server or with the mirror server reaching the afs backend, i'll have to take a closer look when i get a moment | 13:13 |
| mnasiadka | thanks fungi | 13:13 |
| mnasiadka | So, we should be switching Kolla images to build from master, but we need to switch Neutron to uWSGI | 13:13 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/956785 | 13:13 |
| mnasiadka | The CI will fail now, due to the no_log=True issue - but once we revert I'll kick the tyres again | 13:14 |
| mnasiadka | frickler: would be nice if you could have a look ^^ | 13:14 |
| mnasiadka | #topic Current cycle planning | 13:14 |
| mnasiadka | Basically the most important thing is getting the Neutron uWSGI patch merged and then the Kolla master sources one (https://review.opendev.org/c/openstack/kolla/+/949755) | 13:15 |
| mnasiadka | and then we can move back to sort of stable state with master | 13:15 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:15 |
| mnasiadka | bbezak: did you have any time to have a look in Bridging the gap between community and contributing orgs? | 13:16 |
| mnasiadka | I didn't, because firefighting took over my normal life ;-) | 13:16 |
| bbezak | nope, last weeks I'm doing patches and reviews all day long, sorry | 13:16 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible master: Set default external Let's Encrypt cert server https://review.opendev.org/c/openstack/kolla-ansible/+/957141 | 13:17 |
| fungi | no worries, it was mostly just food for thought, so taking time to think about it is good ;) | 13:17 |
| mmalchuk | o/ | 13:17 |
| bbezak | btw - I didn't do a monthly releases as promised, because of recent "security" change. let's stabilize branches, and then I will raise release change | 13:17 |
| mnasiadka | I'll try to get there next week :) | 13:17 |
| fungi | i'm presenting some similar summaries to horizon later today too | 13:17 |
| mnasiadka | bbezak: yup, mariadb unpins, sec thing and other things that need back ports - and then let's make releases | 13:18 |
| mnasiadka | #topic Open discussion | 13:18 |
| mnasiadka | Anything else from anyone? | 13:18 |
| Vii | Could someone help me run the Barbican Vault test? https://review.opendev.org/c/openstack/kolla-ansible/+/935704 :) | 13:18 |
| bbezak | I'm wondering if we should do openbao instead of hashicorp vault there? | 13:20 |
| mnasiadka | run as in? review your code? | 13:21 |
| mnasiadka | Yeah, openbao would be more open - but we should also rework the other hashicorp vault based job | 13:22 |
| frickler | mnasiadka: where did the failures from https://review.opendev.org/c/openstack/kolla-ansible/+/957027 show? why did the CI not fail on that change itself? | 13:22 |
| Vii | I assume that the text must work for you to accept this change | 13:22 |
| mnasiadka | frickler: it seems the uuid nova_cell obfuscation doesn't show up always, but often - we were just too lucky on that one | 13:22 |
| Vii | test* | 13:22 |
| frickler | oh, heisenlogs, nice | 13:23 |
| bbezak | :D | 13:23 |
| mnasiadka | frickler: https://zuul.opendev.org/t/openstack/build/fbffac28ac74420089eb5657c4ad3975/log/primary/logs/ansible/deploy#33410 | 13:23 |
| Vii | but I don't know how to run it, where does the SCENARIO variable come from | 13:23 |
| mnasiadka | Vii: it comes from zuul.d/base.yaml - it defined per base scenario | 13:23 |
| mnasiadka | *it's | 13:23 |
| Vii | I defined it but there is a test and it is skip | 13:24 |
| Vii | https://zuul.opendev.org/t/openstack/build/e413d504c9cd4345ab31d57eff0d05e9/console | 13:24 |
| mnasiadka | I commented, you have barbican-vault in zuul.d/base.yaml and when: scenario == 'barbican' in tests/run.yml | 13:24 |
| mnasiadka | you need to decide | 13:24 |
| Vii | Run test-barbican-vault.sh script - Skipped | 13:24 |
| mnasiadka | Ansible doesn't read your mind yet | 13:24 |
| Vii | aaaaaaa :D | 13:25 |
| Vii | thx | 13:25 |
| fungi | on the earlier topic of bug https://bugs.launchpad.net/kolla-ansible/+bug/2120302 since the vmt has only recently started to oversee deployment projects we're still somewhat unclear on the degree to which our existing policies and classifications are appropriate | 13:25 |
| fungi | since they were originally focused on issues with api services, libraries and client tooling... one thing i'm wondering is exactly how sensitive is the ansible output from k-a expected to be? has there been a lot of work to scrub it? so you have specific policies around that? | 13:25 |
| frickler | seems it does implement "no_no_logging" by doing "s/no/********/", interesting | 13:26 |
| mnasiadka | fungi: I have an idea we could try to have a check if we leak passwords to any logs, it hasn't been a priority for us until now, but I can see that as being a problem | 13:26 |
| fungi | i know in opendev, for example, our deployment logs of ansible output are treated as dangerous to expose, because we don't know what things random modules or playbooks might dump in there | 13:26 |
| mnasiadka | Basically Ansible logs a lot of things via syslog it seems | 13:27 |
| frickler | fungi: yes, I'd assume the situation for kolla-ansible to be similar | 13:27 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret https://review.opendev.org/c/openstack/kolla-ansible/+/935704 | 13:27 |
| mnasiadka | Anyway, we'll keep a better eye on that now, but sadly there's also a lot of other things we need to do outside of security-aware space :) | 13:27 |
| fungi | like, if nova regurgitates a password or key in its servicve log at any log level above debug, we'd consider that a vulnerability... but deployment tool output doesn't seem like it's necessarily got the same set of risks or user expectations | 13:28 |
| fungi | maybe ansible output in this case is akin to debug logging? | 13:28 |
| fungi | we don't treat sensitive data in debug level logs as a vulnerability that warrants an advisory, just a hardening opportunity that can be cleaned up/improved | 13:29 |
| mnasiadka | well, with the problem caused by merging the no_log=True change - I found out nova-manage prints out rabbitmq and database password in the list-cells subcommand | 13:29 |
| fungi | sounds like a deep rabbit hole, pun intended | 13:29 |
| Vii | Yesterday I wrote about another problem I noticed, where can I write it down? | 13:30 |
| Vii | There's another problem, but it requires deeper analysis and consideration. After adding a new "role" that uses Haproxy or ProxySQL, the Docker services don't restart. This results in an error like "User not in database." And you have to manually restart the Haproxy/SqlProxy service. | 13:30 |
| fungi | anyway, just thinking, if the kolla docs don't already make security recommendations about safe handling of ansible output, that could be a good addition | 13:30 |
| Vii | on an already running openstack - adding a new service | 13:30 |
| frickler | Vii: https://bugs.launchpad.net/kolla-ansible | 13:32 |
| mnasiadka | I guess we’re done | 13:40 |
| mnasiadka | Thanks for coming | 13:40 |
| mmalchuk | mnasiadka thanks | 13:40 |
| mnasiadka | #endmeeting | 13:41 |
| opendevmeet | Meeting ended Wed Aug 13 13:41:09 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-08-13-13.00.html | 13:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-08-13-13.00.txt | 13:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2025/kolla.2025-08-13-13.00.log.html | 13:41 |
| amir58118 | thank you all | 13:41 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Revert "security: hide sensitive auth_password in kolla_container module logs" https://review.opendev.org/c/openstack/kolla-ansible/+/957226 | 14:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Drop docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957228 | 16:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Obfuscate auth_password in docker_common_options https://review.opendev.org/c/openstack/kolla-ansible/+/957266 | 16:07 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "security: hide sensitive auth_password in kolla_container module logs" https://review.opendev.org/c/openstack/kolla-ansible/+/957226 | 21:54 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!