13:00:54 <mnasiadka> #startmeeting kolla
13:00:55 <opendevmeet> Meeting started Wed Aug 13 13:00:54 2025 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:55 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:55 <opendevmeet> The meeting name has been set to 'kolla'
13:00:57 <mnasiadka> #topic rollcall
13:00:58 <mnasiadka> o/
13:01:34 <bbezak> o/
13:01:38 <amir58118> \o
13:01:57 <frickler> o/
13:04:47 <mnasiadka> #topic agenda
13:04:47 <mnasiadka> * CI status
13:04:48 <mnasiadka> * Release tasks
13:04:48 <mnasiadka> * Regular stable releases (first meeting in a month)
13:04:48 <mnasiadka> * Current cycle planning
13:04:49 <mnasiadka> * Additional agenda (from whiteboard)
13:04:49 <mnasiadka> * Open discussion
13:04:51 <mnasiadka> #topic CI status
13:05:51 <mnasiadka> It seems the sec bug merge broke our CI, common_opts no_log=True causes getting **** in some output from kolla_container - currently there's a patch up for revert, checking if it comes out green and fast-merging it
13:06:21 <mnasiadka> Otherwise it was good yesterday I guess, so no other CI issues
13:06:31 <mnasiadka> (other than some rax.ord mirror timeouts)
13:06:37 <mnasiadka> which I already reported on #opendev
13:06:55 <frickler> that's docker.io timeouts very likely
13:07:48 <mnasiadka> Oh ok, maybe we should think about mirroring docker rpms/debs in opendev?
13:07:54 <mnasiadka> (but that doesn't show up that often I guess)
13:08:07 <opendevreview> Merged openstack/kolla stable/2024.1: Revert "mariadb: pin to 10.11.13"  https://review.opendev.org/c/openstack/kolla/+/957182
13:08:22 <fungi> basically nothing on those per-provider "mirrors" is actual copies of anything aside from distro packages. all the other trees are caching proxies to external services like dockerhub or pypi
13:09:07 <fungi> so if dockerhub delays or blocks responses to the proxy, then you'll get an error client-side
13:10:59 <mnasiadka> In https://zuul.opendev.org/t/openstack/build/94be78a6d6e249cca2fb160f5eef3bac/log/primary/logs/ansible/bootstrap-servers#764 - it was during downloading gpg and docker client packages
13:11:02 <mnasiadka> so not really dockerhub
13:11:06 <mnasiadka> but I get the point
13:12:54 <mnasiadka> let's move on
13:12:58 <mnasiadka> #topic Release tasks
13:13:00 <fungi> ah, if it was rpms or debs then that might be a connectivity problem between the client and the mirror server or with the mirror server reaching the afs backend, i'll have to take a closer look when i get a moment
13:13:11 <mnasiadka> thanks fungi
13:13:29 <mnasiadka> So, we should be switching Kolla images to build from master, but we need to switch Neutron to uWSGI
13:13:40 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla-ansible/+/956785
13:14:00 <mnasiadka> The CI will fail now, due to the no_log=True issue - but once we revert I'll kick the tyres again
13:14:09 <mnasiadka> frickler: would be nice if you could have a look ^^
13:14:42 <mnasiadka> #topic Current cycle planning
13:15:14 <mnasiadka> Basically the most important thing is getting the Neutron uWSGI patch merged and then the Kolla master sources one (https://review.opendev.org/c/openstack/kolla/+/949755)
13:15:28 <mnasiadka> and then we can move back to sort of stable state with master
13:15:47 <mnasiadka> #topic Additional agenda (from whiteboard)
13:16:16 <mnasiadka> bbezak: did you have any time to have a look in Bridging the gap between community and contributing orgs?
13:16:25 <mnasiadka> I didn't, because firefighting took over my normal life ;-)
13:16:57 <bbezak> nope, last weeks I'm doing patches and reviews all day long, sorry
13:17:01 <opendevreview> Seunghun Lee proposed openstack/kolla-ansible master: Set default external Let's Encrypt cert server  https://review.opendev.org/c/openstack/kolla-ansible/+/957141
13:17:09 <fungi> no worries, it was mostly just food for thought, so taking time to think about it is good ;)
13:17:33 <mmalchuk> o/
13:17:35 <bbezak> btw - I didn't do a monthly releases as promised, because of recent "security" change. let's stabilize branches, and then I will raise release change
13:17:38 <mnasiadka> I'll try to get there next week :)
13:17:44 <fungi> i'm presenting some similar summaries to horizon later today too
13:18:01 <mnasiadka> bbezak: yup, mariadb unpins, sec thing and other things that need back ports - and then let's make releases
13:18:18 <mnasiadka> #topic Open discussion
13:18:22 <mnasiadka> Anything else from anyone?
13:18:35 <Vii> Could someone help me run the Barbican Vault test? https://review.opendev.org/c/openstack/kolla-ansible/+/935704 :)
13:20:56 <bbezak> I'm wondering if we should do openbao instead of hashicorp vault there?
13:21:10 <mnasiadka> run as in? review your code?
13:22:04 <mnasiadka> Yeah, openbao would be more open - but we should also rework the other hashicorp vault based job
13:22:09 <frickler> mnasiadka: where did the failures from https://review.opendev.org/c/openstack/kolla-ansible/+/957027 show? why did the CI not fail on that change itself?
13:22:13 <Vii> I assume that the text must work for you to accept this change
13:22:41 <mnasiadka> frickler: it seems the uuid nova_cell obfuscation doesn't show up always, but often - we were just too lucky on that one
13:22:53 <Vii> test*
13:23:00 <frickler> oh, heisenlogs, nice
13:23:16 <bbezak> :D
13:23:16 <mnasiadka> frickler: https://zuul.opendev.org/t/openstack/build/fbffac28ac74420089eb5657c4ad3975/log/primary/logs/ansible/deploy#33410
13:23:21 <Vii> but I don't know how to run it, where does the SCENARIO variable come from
13:23:37 <mnasiadka> Vii: it comes from zuul.d/base.yaml - it defined per base scenario
13:23:43 <mnasiadka> *it's
13:24:02 <Vii> I defined it but there is a test and it is skip
13:24:36 <Vii> https://zuul.opendev.org/t/openstack/build/e413d504c9cd4345ab31d57eff0d05e9/console
13:24:40 <mnasiadka> I commented, you have barbican-vault in zuul.d/base.yaml and when: scenario == 'barbican' in tests/run.yml
13:24:42 <mnasiadka> you need to decide
13:24:46 <Vii> Run test-barbican-vault.sh script - Skipped
13:24:55 <mnasiadka> Ansible doesn't read your mind yet
13:25:29 <Vii> aaaaaaa :D
13:25:31 <Vii> thx
13:25:33 <fungi> on the earlier topic of bug https://bugs.launchpad.net/kolla-ansible/+bug/2120302 since the vmt has only recently started to oversee deployment projects we're still somewhat unclear on the degree to which our existing policies and classifications are appropriate
13:25:36 <fungi> since they were originally focused on issues with api services, libraries and client tooling... one thing i'm wondering is exactly how sensitive is the ansible output from k-a expected to be? has there been a lot of work to scrub it? so you have specific policies around that?
13:26:01 <frickler> seems it does implement "no_no_logging" by doing "s/no/********/", interesting
13:26:28 <mnasiadka> fungi: I have an idea we could try to have a check if we leak passwords to any logs, it hasn't been a priority for us until now, but I can see that as being a problem
13:26:32 <fungi> i know in opendev, for example, our deployment logs of ansible output are treated as dangerous to expose, because we don't know what things random modules or playbooks might dump in there
13:27:05 <mnasiadka> Basically Ansible logs a lot of things via syslog it seems
13:27:06 <frickler> fungi: yes, I'd assume the situation for kolla-ansible to be similar
13:27:16 <opendevreview> Piotr Milewski proposed openstack/kolla-ansible master: Added vault support to barbican as back-end secret  https://review.opendev.org/c/openstack/kolla-ansible/+/935704
13:27:57 <mnasiadka> Anyway, we'll keep a better eye on that now, but sadly there's also a lot of other things we need to do outside of security-aware space :)
13:28:05 <fungi> like, if nova regurgitates a password or key in its servicve log at any log level above debug, we'd consider that a vulnerability... but deployment tool output doesn't seem like it's necessarily got the same set of risks or user expectations
13:28:46 <fungi> maybe ansible output in this case is akin to debug logging?
13:29:15 <fungi> we don't treat sensitive data in debug level logs as a vulnerability that warrants an advisory, just a hardening opportunity that can be cleaned up/improved
13:29:17 <mnasiadka> well, with the problem caused by merging the no_log=True change - I found out nova-manage prints out rabbitmq and database password in the list-cells subcommand
13:29:45 <fungi> sounds like a deep rabbit hole, pun intended
13:30:15 <Vii> Yesterday I wrote about another problem I noticed, where can I write it down?
13:30:23 <Vii> There's another problem, but it requires deeper analysis and consideration. After adding a new "role" that uses Haproxy or ProxySQL, the Docker services don't restart. This results in an error like "User not in database." And you have to manually restart the Haproxy/SqlProxy service.
13:30:26 <fungi> anyway, just thinking, if the kolla docs don't already make security recommendations about safe handling of ansible output, that could be a good addition
13:30:54 <Vii> on an already running openstack - adding a new service
13:32:10 <frickler> Vii: https://bugs.launchpad.net/kolla-ansible
13:40:06 <mnasiadka> I guess we’re done
13:40:14 <mnasiadka> Thanks for coming
13:40:24 <mmalchuk> mnasiadka thanks
13:41:09 <mnasiadka> #endmeeting