Friday, 2026-03-27

opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: cyborg: Switch to uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/98147005:03
Viihttps://073ac6abaed0915062af-cfb50cc5198678afd23d9756bd7ac094.ssl.cf1.rackcdn.com/openstack/f4a65051956e4f6f80ab7ff5b29a950e/primary/logs/kolla_configs/kolla-toolbox/07:48
Viiconfig.yaml - working 07:53
ViiNot sure if you want to include this fix in the current release cycle or move it to the next one. If we go with this cycle, I should be able to finish it today.07:53
Viimnasiadka: How would you like this to be implemented, using the approach I proposed, or via Jinja2 templates?07:54
ViiIf you’d prefer not to include this in the current cycle, just let me know. I won’t rush it then 😉07:56
mnasiadkaIf we’re able to support both json and yaml on kolla side and slowly move towards yaml per service in k-a - then probably it’s my preference. I’m fine with merging the Kolla side this cycle - but merging X changes in k-a might not happen (one huge patch in k-a is going to be a headache)08:00
ViiWhat about "file" handling? should we keep it as Jinja2 like it is now, or switch to defaults as I suggested?08:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: neutron: Switch neutron_plugin_agent default to ovn  https://review.opendev.org/c/openstack/kolla-ansible/+/97011408:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Support OVN in auto_configure  https://review.opendev.org/c/openstack/kolla-ansible/+/97025008:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: neutron: Switch neutron_plugin_agent default to ovn  https://review.opendev.org/c/openstack/kolla-ansible/+/97011408:22
blanson[m]goooood morning 08:32
mnasiadkaVii: sorry, just trying to get my brain online - can you give me a link to something that explains what you are asking about? ;-)08:37
Viihttps://review.opendev.org/c/openstack/kolla-ansible/+/98224908:38
Viiansible/roles/common/defaults/main.yml08:38
Viimnasiadka: ^08:38
Viiin such a case, for example, permission can be moved to global because it is repeated everywhere and it is easier to change something globally08:39
mnasiadkaI guess that looks fine, optional probably will be ok08:39
blanson[m]I somehow like this way of doing it 08:39
mnasiadkaAnd merging it with some global common config is also a nice idea08:40
mnasiadkablanson[m]: writing JSON is hard08:40
Viiwhat is the way to upload a patch to k-a?08:40
chembervintmnasiadka: hi, could you please take a look on our discussion? I can add IF, but it seams if we want to fix, we have to fix both c-b and c-v. I've just copied logic from c-v https://review.opendev.org/c/openstack/kolla-ansible/+/98226208:41
blanson[m]yh I was saying the way Vii is doing it in this PoC looks really good08:41
blanson[m]this would also allow as you said merging user config into it easily 08:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: cyborg: Switch to uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/98147008:56
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Test for leaked passwords in syslog  https://review.opendev.org/c/openstack/kolla-ansible/+/98106809:02
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Test for leaked passwords in syslog  https://review.opendev.org/c/openstack/kolla-ansible/+/98106809:03
Viihttps://review.opendev.org/c/openstack/kolla/+/982212 <- simple documentation fix before release ?09:08
Viithe biggest mistake is probably "tools/build.py" :P09:09
opendevreviewVictor Chembaev proposed openstack/kolla-ansible master: Run cinder-backup with ipc_mode: host  https://review.opendev.org/c/openstack/kolla-ansible/+/98226209:16
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227509:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237909:21
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227509:25
opendevreviewWill Szumski proposed openstack/kayobe master: Add mechanism to add custom CAs to trust store  https://review.opendev.org/c/openstack/kayobe/+/98181109:26
opendevreviewMerged openstack/kolla-ansible master: cyborg: Switch to uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/98147009:27
opendevreviewMerged openstack/kolla-ansible master: cloudkitty: Switch to uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/97251209:28
*** jhorstmann is now known as Guest598609:37
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227509:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237909:43
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227510:16
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227510:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Test for leaked passwords in syslog  https://review.opendev.org/c/openstack/kolla-ansible/+/98106810:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602110:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602110:58
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602111:00
Viiyesterday RabbitMQ 4.3.0-rc.0 v4.3.0-rc.0  ;) 11:00
ViiSince only 4.2.x clusters can upgrade to 4.3.0 11:01
opendevreviewMerged openstack/kolla-ansible master: RabbitMQ: document handling stream replicas  https://review.opendev.org/c/openstack/kolla-ansible/+/97321211:07
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224911:28
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224911:30
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Block until timedatectl status show synchronized  https://review.opendev.org/c/openstack/kayobe/+/98239611:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602111:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602111:58
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Test for leaked passwords in syslog  https://review.opendev.org/c/openstack/kolla-ansible/+/98106812:11
opendevreviewPierre Riteau proposed openstack/kayobe master: Bump all Ansible collections and roles  https://review.opendev.org/c/openstack/kayobe/+/98239912:11
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237912:59
Vii++ https://review.opendev.org/c/openstack/kolla-ansible/+/966021 ?13:05
Viijob SLURP broken13:05
Viihttps://zuul.opendev.org/t/openstack/build/5825f239ef614b639df92d4451957dca13:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache  https://review.opendev.org/c/openstack/kolla-ansible/+/98241413:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache  https://review.opendev.org/c/openstack/kolla-ansible/+/98241413:19
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache  https://review.opendev.org/c/openstack/kolla-ansible/+/98241413:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237913:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache  https://review.opendev.org/c/openstack/kolla-ansible/+/98241413:28
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Test for leaked passwords in syslog  https://review.opendev.org/c/openstack/kolla-ansible/+/98106813:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237913:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for running WSGI under Apache  https://review.opendev.org/c/openstack/kolla-ansible/+/98241413:57
mnasiadkaVii: well, the upgrade job in 966021 should run code from master branch, but for some reason it’s not doing that?13:59
mnasiadkaVii: ah, that’s old incarnation, current run works fine14:00
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602114:00
tafkamaxmnasiadka: I see you are doing ze wsgi apache removal. If I wish to backport the 975901 fix then It would require to verify apache works, as you don't have any plans in backporting the apache removal?14:02
tafkamaxIs there a way to run CI under 2025.2 aswell?14:02
Viiah14:03
mnasiadkatafkamax: just make a cherry pick of your patch to 2025.2 to check if your patch works?14:03
mnasiadkatafkamax: I was planning to wait until we merge your patch with the removal14:03
tafkamaxTo do that in CI, it is as simple as git checkout 2025.2-stable and then git checkout -b bug/XXXXX and then add the cherry pick?14:03
Viihttps://review.opendev.org/c/openstack/kolla-ansible/+/982249 Will you check if it is accepted? The tests pass.14:04
tafkamaxok thanks for the info14:04
Viihttps://zuul.opendev.org/t/openstack/status/pipeline/check?&change=98224914:04
Viibecause if you want, I can do the rest quickly in this cycle, because later there might be no time14:04
mnasiadkatafkamax: you can cherry pick using Gerrit UI14:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-logs: break out from common role  https://review.opendev.org/c/openstack/kolla-ansible/+/96330314:07
mnasiadkaVii: We wanted to add YAML to set_configs without removing JSON support and remove JSON support when the switch is done - it’s a bit unrealistic to do that in one cycle, and we need to think about people using Kolla without using Kolla-Ansible - we need to deprecate JSON support and wait until the next SLURP cycle to remove it14:10
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318814:11
opendevreviewTaavi Ansper proposed openstack/kolla-ansible master: Fix keystone with IDP configured.  https://review.opendev.org/c/openstack/kolla-ansible/+/97590114:12
tafkamaxok, will try to test for apache14:12
ViiI understand the concern regarding external users and the SLURP cycle policy. My goal was to avoid configuration duality, but I agree that a more gradual transition is safer.14:14
ViiI propose to keep JSON support in kolla_set_configs (which we already do in the current hybrid script).14:15
ViiModify the Ansible task to NOT delete config.json for now. Instead, we can generate config.yaml alongside it or keep using JSON if the operator prefers.14:15
tafkamaxDid I remove too many zuul jobs? https://zuul.opendev.org/t/openstack/status?change=97590114:15
ViiAdd a formal deprecation warning in the release notes, stating that config.json support is deprecated and will be removed in the 2027.1 cycle.14:16
tafkamaxit is in queue, or is this ok?14:16
ViiThis way, we provide the YAML feature now without breaking third-party workflowss14:16
tafkamaxok it started now, phew14:19
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224914:21
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224914:22
tafkamaxI have to say I do like ze yaml structure alot. I try to use variable | combine as much as possible :)14:24
tafkamaxand lists are banned, use dicts instead for better "mergeability"14:24
opendevreviewPiotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine  https://review.opendev.org/c/openstack/kolla/+/98227514:33
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224914:34
ViiI've updated the approach to follow the deprecation policy safely. Both config.json and config.yaml are now generated if sources exist. To prevent containers from using outdated JSON after upgrade, I've updated kolla_set_configs to prioritize YAML in the search order. This ensures the new config is used when available while maintaining full  for JSON-only users.14:37
ViiLet me know what you think of this approach. If you're still not sure and would prefer not to push this in the current cycle, that's completely fine with me. However, if it looks good to you, please let me know what needs further adjustment. I'm ready to apply this pattern to the remaining roles once we agree on the final logic.14:38
Viimnasiadka: ^14:39
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: common: migrate container configuration to YAML structures  https://review.opendev.org/c/openstack/kolla-ansible/+/98224914:44
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237914:58
opendevreviewOwen Jones proposed openstack/kayobe master: Build Rocky based IPA images  https://review.opendev.org/c/openstack/kayobe/+/98210915:07
tafkamaxBrr mnasiadka did you remove mod auth_openidc_module ?15:08
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Block until timedatectl status show synchronized  https://review.opendev.org/c/openstack/kayobe/+/98239615:08
tafkamaxI am getting Invalid command 'OIDCHTMLErrorTemplate', perhaps misspelled or defined by a module not included in the server configuration15:08
tafkamaxhttps://opendev.org/openstack/kolla/src/commit/15c4e4d41665b06b1e29ae200e1d0171f14cd14b/docker/keystone/keystone-base/Dockerfile.j2#L2715:11
tafkamaxseems to be present15:11
tafkamax(master link) https://opendev.org/openstack/kolla/src/branch/master/docker/keystone/keystone-base/Dockerfile.j2#L2715:11
tafkamaxHmm how can I add apachectl -M to debug.15:16
mnasiadkatafkamax: https://github.com/OpenIDC/mod_auth_openidc/blob/ed819f1074107eed642f10d30e801a11bc8ce030/ChangeLog#L900 - seems related15:18
tafkamaxwow good find15:18
tafkamax- remove support for OIDCHTMLErrorTemplate, deprecated since 2.4.1415:20
tafkamaxBut in keystone-httpd it is still working though.15:23
tafkamaxLog file btw: https://1dfa64b9a521e63098e8-48d863b7f9e161f294d24ab8665672f8.ssl.cf1.rackcdn.com/openstack/50a03f758f854693945f579707818490/primary/logs/container_logs/keystone.txt15:25
mnasiadkaWell, it depends what mod_auth_openidc version is in the rpm/debs that we install15:27
tafkamaxYeah, I can see that noble is still running.15:27
tafkamaxtrixie and rocky10 failed15:28
tafkamaxOh boy, so it would need rework :D15:28
mnasiadkaSo, maybe we need to post a patch to support it only on noble?15:29
mnasiadkaAnd backport it all the way?15:29
mnasiadkaOf course with some precheck and a reno15:30
mnasiadkaAnd then remove that option in master15:30
mnasiadkacompletely15:30
mnasiadkaI know it’s a lot of work though :(15:30
tafkamaxI am looking at the alternative. "rely on standard Apache error handling capabilities by default15:31
tafkamax  environment variable strings REDIRECT_OIDC_ERROR and REDIRECT_OIDC_ERROR_DESC"15:31
tafkamaxSo these variables are available in default error page then?15:31
tafkamaxatleast in noble using wsgi_provider: apache works.15:35
tafkamaxhttps://zuul.opendev.org/t/openstack/build/912324a27a104e98ba4c7382fffaa56515:35
tafkamaxhttps://github.com/OpenIDC/mod_auth_openidc/discussions/107215:39
tafkamaxhttps://github.com/OpenIDC/mod_auth_openidc/wiki#19-how-do-i-customize-the-user-facing-error-pages15:41
tafkamaxI am thinking what if we migrate to this new version?15:42
tafkamaxis it too much to backport?15:42
tafkamaxLooking at an prod-cluster 2025.2 keystone-httpd container there is no reference to error template15:45
tafkamaxinteresting15:46
tafkamaxalso in 2025.1 cluster with keystone container15:48
tafkamaxweird, my prod clusters don't have that :/15:48
tafkamaxok, so in 2025.1 prod cluster this option is not present indeed: https://opendev.org/openstack/kolla-ansible/src/branch/stable/2025.1/ansible/roles/keystone/templates/wsgi-keystone.conf.j215:57
tafkamaxIn 2025.2 it is present, but in 2025.2 cluster I am already using keystone-httpd15:57
tafkamaxI think adding custom error htmls is a big hassle, what do yo uthink?16:01
tafkamaxUsing ansible to "inject" lineinfile with the new variables to /usr/share/apache2/error, does this feel like too much of an hack?16:04
tafkamaxIssue comes from here btw16:07
tafkamaxhttps://review.opendev.org/c/openstack/kolla-ansible/+/83280616:07
tafkamaxI don't think this has been backported to 2025.116:08
tafkamaxIt truly is ironic, the patch was initally made in 2022, then it took years to get it merged and by that time kolla changed to uwsgi and the option has been deprecated and removed.16:09
tafkamaxTBH I get why the template was made, it matches all errors and then does the redirect. The new logic would be to create a similar page, but just use ErrorDocument ?16:15
tafkamaxOr if the REDIRECT_OIDC_ERROR is set we can have a conditional in /usr/share/apache2/error ? IMHO we don't want to edit /usr/share/apache2/error files.16:16
tafkamaxI am just thinking out loud.16:16
tafkamaxYeah AI also said not to edit in /usr/share as that is package manager world.16:20
tafkamaxAnd the guy who made the Patch has apache.org in his email, cant make this joke up 🤣16:22
tafkamaxI am thinking that if I don't know what Errorcode to match then I don't know what ErrorDocument to use in this case. Does the issue arise with 400,401,404 or 5XX series of errors?16:25
tafkamaxOk so I am looking at ze patch and it says "authentication error timeout page"16:34
tafkamaxWould it be too wild of an guess to just say ErrorDocument 401 /var/www/html/modoidc-error-page.html and then use the variables referenced in https://github.com/OpenIDC/mod_auth_openidc/wiki#19-how-do-i-customize-the-user-facing-error-pages ?16:35
opendevreviewTaavi Ansper proposed openstack/kolla-ansible master: Fix keystone with IDP configured.  https://review.opendev.org/c/openstack/kolla-ansible/+/97590116:54
opendevreviewHao Wang proposed openstack/kolla-ansible master: Support deploy Zaqar with kolla-ansible  https://review.opendev.org/c/openstack/kolla-ansible/+/89261517:16
opendevreviewHao Wang proposed openstack/kolla master: Support building zaqar container image in Kolla  https://review.opendev.org/c/openstack/kolla/+/88756017:22
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237917:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237917:42
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kolla-logs: break out from common role  https://review.opendev.org/c/openstack/kolla-ansible/+/96330317:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Rename role to kolla-toolbox  https://review.opendev.org/c/openstack/kolla-ansible/+/96318817:48
tafkamaxmnasiadka: well I got it to work in trixie and rocky failed spuriously on nova-scheduler.18:41
tafkamaxI will recheck for all three18:41
tafkamaxWhat do you think of my solution? Or should I do something differently?18:41
tafkamaxFor trixie logs are here: https://zuul.opendev.org/t/openstack/build/bb11ce415afc4d29a4250ba4f041545c/logs18:42
mnasiadkatafkamax: happy to have a look on Monday, it’s nearly 8pm Friday ;)18:45
tafkamaxsure, yeah18:53
opendevreviewMerged openstack/kolla-ansible master: kolla-ansible: Add RabbitMQ 4.2 support and dynamic upgrade paths  https://review.opendev.org/c/openstack/kolla-ansible/+/96602119:17
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add uWSGI support for prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/98237919:30

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!