| opendevreview | Michael Still proposed openstack/kolla-ansible master: Allow requiring secure channels with SPICE https://review.opendev.org/c/openstack/kolla-ansible/+/967802 | 01:06 |
|---|---|---|
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Deploy Kerbside with Kolla-Ansible. https://review.opendev.org/c/openstack/kolla-ansible/+/976889 | 01:06 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Add kerbside CI scenario jobs. https://review.opendev.org/c/openstack/kolla-ansible/+/988189 | 01:06 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Allow requiring secure channels with SPICE https://review.opendev.org/c/openstack/kolla-ansible/+/967802 | 04:43 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Deploy Kerbside with Kolla-Ansible. https://review.opendev.org/c/openstack/kolla-ansible/+/976889 | 04:43 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Add kerbside CI scenario jobs. https://review.opendev.org/c/openstack/kolla-ansible/+/988189 | 04:43 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Allow requiring secure channels with SPICE https://review.opendev.org/c/openstack/kolla-ansible/+/967802 | 05:38 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Deploy Kerbside with Kolla-Ansible. https://review.opendev.org/c/openstack/kolla-ansible/+/976889 | 05:38 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Add kerbside CI scenario jobs. https://review.opendev.org/c/openstack/kolla-ansible/+/988189 | 05:38 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/2026.1: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988203 | 05:56 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/2025.2: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988209 | 07:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/2025.2: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988209 | 08:39 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Allow requiring secure channels with SPICE https://review.opendev.org/c/openstack/kolla-ansible/+/967802 | 08:51 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Deploy Kerbside with Kolla-Ansible. https://review.opendev.org/c/openstack/kolla-ansible/+/976889 | 08:51 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Add kerbside CI scenario jobs. https://review.opendev.org/c/openstack/kolla-ansible/+/988189 | 08:51 |
| *** jhorstmann is now known as Guest9266 | 10:00 | |
| opendevreview | Merged openstack/kayobe master: Fix network connectivity check on modern ansible https://review.opendev.org/c/openstack/kayobe/+/986126 | 10:11 |
| opendevreview | Merged openstack/kayobe master: Register overcloud hosts in Bifrost https://review.opendev.org/c/openstack/kayobe/+/936032 | 10:26 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2026.1 failed: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988203 | 11:40 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Add kayobe-ansible-control-host-configure jobs https://review.opendev.org/c/openstack/kayobe/+/972843 | 12:33 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix deprecated Templar attribute access in plugins https://review.opendev.org/c/openstack/kayobe/+/986330 | 13:26 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix string-typed conditional deprecation warnings https://review.opendev.org/c/openstack/kayobe/+/986370 | 13:26 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix reserved variable names in serial console playbook https://review.opendev.org/c/openstack/kayobe/+/986332 | 13:26 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix deprecated play_hosts in network connectivity https://review.opendev.org/c/openstack/kayobe/+/986371 | 13:26 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2026.1 failed: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988203 | 13:52 |
| opendevreview | Verification of a change to openstack/kayobe master failed: CI: Add kayobe-ansible-control-host-configure jobs https://review.opendev.org/c/openstack/kayobe/+/972843 | 15:49 |
| blanson[m] | hello we're seeing an issue in k-a idk if it's been fixed already or not (2025.1), but the user creation process isn't idempotent, which invalidates every single token when it happens. mnasiadka are you aware of this or is it a new thing ? | 15:59 |
| mnasiadka | I think I’m aware, but I don’t know if we can do anything about it | 16:00 |
| blanson[m] | hm | 16:00 |
| mnasiadka | https://opendev.org/openstack/kolla-ansible/src/commit/e302bab31f30296f7d6f41c3d46e19cf5668b97a/ansible/roles/service-ks-register/tasks/main.yml#L82 | 16:00 |
| blanson[m] | I'll try myself a patch tonight cause it's fairly critical for us, nothing can get reconfigured without massive downtime on our ceph clusters cause we have to restart ever y 250ish RGW | 16:01 |
| mnasiadka | You can set update_keystone_service_user_passwords to false | 16:01 |
| blanson[m] | oih | 16:01 |
| blanson[m] | oh | 16:01 |
| blanson[m] | I missed it | 16:01 |
| tafkamax | Hmm, how would one experience this? | 16:02 |
| blanson[m] | well this will be a nice workaround for now | 16:02 |
| tafkamax | Deploy or reconfig? | 16:02 |
| tafkamax | Or in both? | 16:02 |
| blanson[m] | Taavi Ansper: both | 16:02 |
| mnasiadka | Deploy == reconfig | 16:02 |
| mnasiadka | (In terms of kolla-ansible subcommands) | 16:02 |
| blanson[m] | this would need to be fixed in ansible-collection-openstack I assume ? | 16:02 |
| mnasiadka | Yeah, but how? | 16:02 |
| mnasiadka | We can’t fetch the password and compare it, can we? | 16:03 |
| blanson[m] | yh idk either | 16:03 |
| mnasiadka | blanson[m]: are you using service tokens for RGW, or just user/pass? | 16:03 |
| blanson[m] | service tokens | 16:03 |
| blanson[m] | actually it's a bit more complicated | 16:03 |
| blanson[m] | we use both tgw s3 api and swift compatibility api | 16:03 |
| blanson[m] | s3 api works fine on reconfigure | 16:04 |
| blanson[m] | but the swift api is bugged | 16:04 |
| blanson[m] | gets a 401 from keystone because it expires a token | 16:04 |
| blanson[m] | and never tries to renew its token | 16:04 |
| mnasiadka | Maybe it’s a keystone bug? | 16:04 |
| tafkamax | Yeah password related checka are a nuisance | 16:04 |
| mnasiadka | I mean changing the keystone user password should not make the tokens invalidated? | 16:04 |
| mnasiadka | Or maybe an RGW bug that it should re-authenticate? | 16:05 |
| blanson[m] | it's changing the ceph_rgw user password | 16:24 |
| blanson[m] | that discards all the tokens | 16:24 |
| tafkamax | One way would be to try authenticating with the password to umm test if auth works. If it does, don't change the pw... | 16:28 |
| mnasiadka | Well, authenticating to a user can enable some red flags in some security monitoring systems :) | 16:28 |
| tafkamax | There is always a catch22 | 16:31 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Add kayobe-ansible-control-host-configure jobs https://review.opendev.org/c/openstack/kayobe/+/972843 | 16:39 |
| opendevreview | Merged openstack/kolla-ansible stable/2026.1: Fix for overly restrictive permissions on ca-certificates directory https://review.opendev.org/c/openstack/kolla-ansible/+/988203 | 17:10 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Only remove loopback record if not mapped to IP https://review.opendev.org/c/openstack/kayobe/+/988138 | 17:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test all jobs https://review.opendev.org/c/openstack/kolla-ansible/+/988307 | 17:37 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Only remove loopback record if not mapped to IP https://review.opendev.org/c/openstack/kayobe/+/988138 | 17:40 |
| blanson[m] | to be fair I think the parameter for changing or not the passwords on every apply is a nice compromise. I didn't know it existed so yesterday at 3AM was fun, but we'll switch it to false, and this should be good enough, you can always enable it temporarily to rotate passwords and off you go | 17:57 |
| tafkamax | Maybe update docs to bring this option out? | 17:59 |
| tafkamax | And not staying an obscure variable when looking through the code. | 17:59 |
| blanson[m] | true idk where it should go ? | 18:00 |
| blanson[m] | rgw doc maybe ? that's the most likely to break | 18:00 |
| tafkamax | If it is, i will take back my words. | 18:00 |
| tafkamax | I don't know by heart but maybe someplace where secrets config is explained | 18:00 |
| tafkamax | Could link to RGW setup as a NB! | 18:01 |
| blanson[m] | I'm thinking a warning here https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html#radosgw ? | 18:01 |
| blanson[m] | under keystone section something about advising operators to disable password set always because it invalidates tokens a can cause unforseen outages ? | 18:03 |
| blanson[m] | let me come up with something real quick | 18:03 |
| tafkamax | https://docs.openstack.org/kolla-ansible/latest/reference/shared-services/keystone-guide.html | 18:04 |
| tafkamax | Maybe here? | 18:04 |
| tafkamax | Yes, as you said, under keystone | 18:04 |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: docs: warn about password resets in ceph RGW doc https://review.opendev.org/c/openstack/kolla-ansible/+/988321 | 18:13 |
| blanson[m] | let me look at what you sent | 18:14 |
| blanson[m] | I don't know if it's really a keystone issue tho ? | 18:17 |
| blanson[m] | while we're at it we could add it somewhere else | 18:21 |
| blanson[m] | > I mean changing the keystone user password should not make the tokens invalidated? | 18:22 |
| blanson[m] | mnasiadka that is also true I shall probbly ask the keystone people about it | 18:22 |
| blanson[m] | > Or maybe an RGW bug that it should re-authenticate? | 18:24 |
| blanson[m] | this is what we discovered in the ceph code this morning, it has retry logic for s3, but not swift for some reason, because it assumes 401 is invalid signature for the client token, but it could be rgw's own token aswell, and it can't make a difference apparently | 18:24 |
| tafkamax | I was thinking it is a keystone docs addition in kolla-ansible docs, because it deals with configuring users authentication, which is the role of keystone in openstack - authenticating services. So if this applies to all users in keystone, when using kolla-ansible, then it should be under the central location - keystone. | 18:29 |
| blanson[m] | that makes sense, I shall add a note in keystone aswell. I still think the ceph RGW part is important, cause I figure most people won't read the keystone part to in-depth as it's pretty automagic in k-a ? | 18:31 |
| blanson[m] | also keystone peeps told me it was 100% expected behavior from keystone, I don't really know why but it seems to be by design that it revokes past tokens on password set | 18:31 |
| tafkamax | I agree, but it would be the correct place where to put it. Otherwise it would be confusing with a warning about passwords being in a completly different place. I agree that it is OK to add a link to this warning in RGW. As I understand it affects different services differently. | 18:33 |
| tafkamax | We have done quite a few upgrades and haven't explicilty seen this behavior, tho we use just ceph rbd and not rgw. | 18:33 |
| blanson[m] | it's very noticeable on rgw cause most object storage consumers exclusively use tokens | 18:35 |
| blanson[m] | on rbd you shouldn't have to worry about it cause it's mostly the other way around ? openstack authenticate to ceph | 18:35 |
| tafkamax | yes | 18:35 |
| blanson[m] | whereas RGw authenticate to keystone on behalf of the customer | 18:35 |
| tafkamax | you put the cephx keyring in openstack config | 18:36 |
| tafkamax | oh oka | 18:36 |
| tafkamax | * oh okay | 18:36 |
| mnasiadka | blanson[m]: I would assume that for ceph_rgw we might just set update_password: on-create? | 18:40 |
| mnasiadka | Although maybe the default should be on-create and we should document that if you want to rotate passwords - you set this variable to true so it updates your services passwords? | 18:40 |
| mnasiadka | Ideally maybe we have a subcommand to rotate passwords? | 18:41 |
| blanson[m] | also true | 18:41 |
| blanson[m] | this would make one more task idempotent | 18:41 |
| tafkamax | If a service appears or disappears would you want to set it to true? | 18:41 |
| mnasiadka | (That clears out passwords in password.yml, runs kolla-genpwd and updates the passwords) | 18:41 |
| blanson[m] | on our eternal quest to full idempotence | 18:41 |
| tafkamax | E.g. when doing a upgrade between releases or adding services or removing them? | 18:42 |
| blanson[m] | Taavi Ansper: on_create would still create appearing services | 18:42 |
| mnasiadka | tafkamax: well, if you’re deploying a new service - the user should not exist :) | 18:42 |
| mnasiadka | And on-create would still create | 18:42 |
| mnasiadka | Well, create/update password, the user will get created always :) | 18:42 |
| blanson[m] | removal should be an operator process imo, although we coudl facilitate that in some capacity ? | 18:42 |
| mnasiadka | blanson[m]: maybe, but that’s long term thinking | 18:43 |
| blanson[m] | yh that's me being unrealistic on how much time I have in day | 18:43 |
| blanson[m] | I will bring up the point of changing the default behavior for the pasword_set tmrw during weekly to get every1's position on it | 18:46 |
| mnasiadka | Good idea | 19:04 |
| mnasiadka | blanson[m]:add that to additional agenda | 19:04 |
| blanson[m] | ayyy that's right we have that | 19:06 |
| kevko | hmm, am I removed from cores ? I can't vote for +w | 19:41 |
| tafkamax | Yes, it was in openstack-discuss mailing list some time ago... | 19:44 |
| kevko | tafkamax: Ah, I see now, I understand - it's true that lately I’ve been allocated by my company to operations work, so I’ve had little time for Kolla and its development itself, but that should hopefully change in the foreseeable future, so hopefully I’ll earn my membership back :) . thanks tafkamax for pointing me to the right channel | 19:50 |
| tafkamax | Yeah, understandable! | 19:54 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Fix D001 Line too long errors in release notes https://review.opendev.org/c/openstack/kolla/+/988347 | 19:59 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix deprecated Templar attribute access in plugins https://review.opendev.org/c/openstack/kayobe/+/986330 | 20:23 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Add support for rocky-security.repo https://review.opendev.org/c/openstack/kayobe/+/988354 | 21:32 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Ignore D001 Line too long errors in release notes https://review.opendev.org/c/openstack/kolla/+/988347 | 21:48 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Prepare /var/log/kolla in the base start script https://review.opendev.org/c/openstack/kolla/+/985300 | 21:49 |
| opendevreview | Michal Arbet proposed openstack/kolla master: fluentd: add remote_syslog output plugin support https://review.opendev.org/c/openstack/kolla/+/984472 | 21:49 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2025.2 failed: Fix ulimit defaults for Debian family container engines https://review.opendev.org/c/openstack/kolla-ansible/+/987831 | 21:58 |
| *** Viii6 is now known as Viii | 22:43 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!