| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 980185: WIP Elements to use bootc container images for root content https://review.opendev.org/c/openstack/diskimage-builder/+/980185 | 03:11 | |
| @mnasiadka:matrix.org | Well, maybe we should investigate Anubis then, because it seems to be more and more of a problem. | 06:07 |
|---|---|---|
| @fungicide:matrix.org | #status log Restarted Mailman and related services on lists01 in order to relieve memory pressure | 12:31 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 12:31 |
| @capt123:matrix.org | Hey guys is this a right place to ask question regarding openstack ? | 12:43 |
| @fungicide:matrix.org | Sahil Kumar: not really, no. openstack has a bunch of specific irc channels, as well as a matrix room for operators and mailing lists, depending on what you're needing to talk about | 12:57 |
| @fungicide:matrix.org | are you using openstack in someone's cloud, installing/running openstack yourself, trying to contribute to openstack development, or something else? | 12:58 |
| @fungicide:matrix.org | https://docs.openstack.org/contributors/ | 12:59 |
| @capt123:matrix.org | So we are moving to on-prem and basically we process 450M+ transactions daily want to offload to on-prem exploring ways We are currently working on designing credential/secret management for our platform built on OpenStack, and we are trying to reason about it from first principles instead of assuming the default OpenStack approach. | 13:00 |
| Our requirement is tenant-level secure credential storage (similar to a KMS) where identities/secrets are not stored directly on disk. | ||
| We looked at Barbican, but from our understanding it does not fully behave like a tenant-scoped credential store in the way a typical KMS does. | ||
| Questions: | ||
| Has anyone implemented tenant-level secret storage / credential management on top of OpenStack? | ||
| Did you extend Barbican, integrate an external KMS (like Hashicorp Vault), or use some other pattern? | ||
| How do you avoid storing sensitive identities or credentials directly on disk? | ||
| Appreciate any thoughts or references. Thanks! | ||
| @capt123:matrix.org | fungi: if this is not a right just help me to navigate to a correct channel | 13:05 |
| @capt123:matrix.org | * fungi: if this is not a right channel just help me to navigate to a correct channel | 13:05 |
| @tafkamax:matrix.org | #openstack-barbican on OFTC. | 13:06 |
| @tafkamax:matrix.org | maybe they have ideas? | 13:06 |
| @fungicide:matrix.org | Sahil Kumar: there is a #openstack-ops:opendev.org matrix room where people discuss installing and running openstack, and also the https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/ is a popular place to ask questions about those sorts of topics | 13:14 |
| @fungicide:matrix.org | as i said, this is not an appropriate place to ask questions about openstack, they're a separate community | 13:14 |
| @capt123:matrix.org | Sure thanks for the help fungi | 13:15 |
| @fungicide:matrix.org | i was just trying to figure out what kind of question you had so i could steer you to the right part of the openstack community | 13:15 |
| @fungicide:matrix.org | i'm going to disappear for a bit to get lunch, back shortly | 15:00 |
| @clarkb:matrix.org | I'm having a bit of a slow start this morning. But my intention is to approve https://review.opendev.org/c/opendev/system-config/+/978980 once I'm able to sit at a computer for a consistent period of time. mnasiadka then I guess the next step would be to sync up and walk through some things. Do you have time late tomorrow afternoone your time / early morning my time either tomorrow, monday, or wednesday? I've got meetings all tuesday so that isn't a good day | 15:11 |
| @clarkb:matrix.org | then if others have time to review my gerrit upgrade test fix change and its child that adds gerrit 3.13 images and testing that would be a big help to getting the gerrit upgrade process moving forward | 15:24 |
| @clarkb:matrix.org | ok I think I am ready to approve 978980 now. fungi when you get back from lunch is there any reason not to do that? I think things are mostly stable this morning? | 16:04 |
| @clarkb:matrix.org | The puppet job is still failing I bet but that shouldn't impact this change | 16:14 |
| @fungicide:matrix.org | i'm already +2 on 978980 so go ahead as far as i'm concerned | 16:33 |
| @clarkb:matrix.org | ok done | 16:43 |
| @fungicide:matrix.org | thanks! | 16:45 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 978980: Add Michal Nasiadka to base_users on all hosts https://review.opendev.org/c/opendev/system-config/+/978980 | 17:19 | |
| @clarkb:matrix.org | cool now for that to deploy. mnasiadka definitely let me know when is good for you and we'll sync up together and work through some of the boostrapping stuff | 17:21 |
| @mnasiadka:matrix.org | Clark: sorry, I've seen your message but got overwhelmed by some other work - your early morning tomorrow or on Monday is fine :) | 17:22 |
| @clarkb:matrix.org | mnasiadka: ok why don't see see how things are tomorrow at about 1500 UTC and if you or I are busy again we'll look at Monday 1500 UTC | 17:22 |
| @fungicide:matrix.org | don't feel bad for getting distracted, we all do it. i'm distracted by other things at this very moment, in fact | 17:23 |
| @mnasiadka:matrix.org | I don't feel bad, don't worry - but maybe Clark is feeling bad with my delay :D | 17:24 |
| @fungicide:matrix.org | i doubt it | 17:24 |
| @clarkb:matrix.org | nah its fine. I too have things to be distracted by | 17:24 |
| @fungicide:matrix.org | we're all drowning in a sea of $other | 17:25 |
| @clarkb:matrix.org | like expensing that wiki cert | 17:25 |
| @clarkb:matrix.org | infra-prod-base failed so I'm looking at that now | 17:30 |
| @clarkb:matrix.org | gitea10 failed because it couldn't get the apt/dpkg lock. This doesn't appear to have caused any further fallout so I think we can just let this be and let the daily runs get gitea10 up to date | 17:32 |
| @fungicide:matrix.org | yeah, probably just collided with a random package update run | 17:45 |
| @fungicide:matrix.org | if we really wanted, we could disable automated update services from cron and directly run unattended-upgrades on every server in our periodic buildset | 17:46 |
| @fungicide:matrix.org | daily periodic i mean | 17:47 |
| @clarkb:matrix.org | oh that is an idea. The downside to thati s when we have hosts in the emergency file (like static right now?) we'd stop updated for a bit | 17:54 |
| @clarkb:matrix.org | fungi: should we start approving some of those changes for static? I expect to be around today and can help monitor | 17:54 |
| @clarkb:matrix.org | also if you have time for https://review.opendev.org/c/opendev/system-config/+/979874/ and child that would be a help on the gerrit upgrade front | 17:55 |
| @fungicide:matrix.org | yeah i'm around all day if you want to approve waf changes | 17:56 |
| @fungicide:matrix.org | approved 979874 for the gerrit upgrade testing fix, good find | 17:57 |
| @fungicide:matrix.org | child is taking me a little longer | 17:59 |
| @clarkb:matrix.org | fungi: is https://review.opendev.org/c/opendev/system-config/+/979089 the first one we should approve so that we can remove the emergency file entry for static02? | 18:01 |
| @fungicide:matrix.org | yes, ideally we take it out of the emergency disable list before that deploys | 18:05 |
| @fungicide:matrix.org | i can do that once it's approved | 18:05 |
| @clarkb:matrix.org | ok I'll approve that one now first then we can followup with the others once emergency file is back to normal etc | 18:05 |
| @fungicide:matrix.org | and static02 is now out of the disable list after 9 days | 18:09 |
| @clarkb:matrix.org | I'm actually suddenly concerned that https://review.opendev.org/c/opendev/system-config/+/979875 won't publish the gerrit 3.13 images | 18:12 |
| @clarkb:matrix.org | I don't think that is a major issue but let me look at file matchers | 18:12 |
| @clarkb:matrix.org | we modify testinfra/test_gerrit.py which is in the list of files so actually i think this is fine. And if it isn't fine we can followup with a fix for whatever does go wrong there. We will rebuild our prod image but since we're using fixed tags now I don't feel like we need to restart on the new image as it should functiaonlly be a noop | 18:14 |
| @clarkb:matrix.org | fungi: ^ just a heads up and feel free to disagree with this assessment if you don't think that is safe | 18:15 |
| @fungicide:matrix.org | no i agree it's fine | 18:17 |
| @fungicide:matrix.org | the new image should be functionally equivalent to the old | 18:17 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 979874: Actually test Gerrit upgrade from 3.11 to 3.12 https://review.opendev.org/c/opendev/system-config/+/979874 | 18:26 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 979089: Add WAF rules for docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/979089 | 18:44 | |
| @clarkb:matrix.org | deployment is in progress | 18:47 |
| @clarkb:matrix.org | fungi: it looks like the vhost config is correct after deployment | 18:50 |
| @clarkb:matrix.org | did you want to double check before I start approving some of the other waf changes? | 18:50 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/978110 and https://review.opendev.org/c/opendev/system-config/+/979090/ and https://review.opendev.org/c/opendev/system-config/+/978111 should all be safe followups now I think | 18:51 |
| @clarkb:matrix.org | actually https://review.opendev.org/c/opendev/system-config/+/979090 has an unmerged depends on so probably just the other two | 18:54 |
| @clarkb:matrix.org | I've gone ahead and approved them so that I can grab lunch while we wait | 18:56 |
| @fungicide:matrix.org | looking | 18:57 |
| @fungicide:matrix.org | updated vhost config looks right and server-status indicates apache is still happy | 18:59 |
| @fungicide:matrix.org | error log shows new matches are happening too | 18:59 |
| @fungicide:matrix.org | uri "/developer/diskimage-builder/user_guide/elements/iscsi-boot/developer/elements/disable-selinux/elements/deploy-tgtadm/developer/elements/fips/elements/sysprep/README.html" | 19:03 |
| @fungicide:matrix.org | yeah nah | 19:03 |
| @fungicide:matrix.org | ever since things stabilized from the disruption last week, i haven't seen nf_conntrack_count climb above 25% of the default max, so i continue to think we don't need to adjust it permanently | 19:11 |
| @clarkb:matrix.org | Makes sense and better to have the early warning signal probably | 19:12 |
| @fungicide:matrix.org | agreed, i've lowered it back fro 524288 to 65536 now to match what it would otherwise be after the next reboot | 19:13 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 979875: Add Gerrit 3.13 images and testing https://review.opendev.org/c/opendev/system-config/+/979875 | 19:23 | |
| @clarkb:matrix.org | It did promote all three images just fyi so I think all is as expected | 19:30 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: | 19:32 | |
| - [opendev/system-config] 978110: Add WAF rules to more static sites https://review.opendev.org/c/opendev/system-config/+/978110 | ||
| - [opendev/system-config] 978111: Test that WAF rules share a common block pool https://review.opendev.org/c/opendev/system-config/+/978111 | ||
| @clarkb:matrix.org | fungi: were there any more waf changes that weren't waiting on parents in other repos to merge first? | 19:35 |
| @fungicide:matrix.org | i don't think so | 19:42 |
| @fungicide:matrix.org | https://review.opendev.org/978956 maybe? | 19:43 |
| @fungicide:matrix.org | i think that's the only one not still blocked, and is a good fix to include | 19:44 |
| @clarkb:matrix.org | That is a good point. We have already applied that to the server and since ansible doesn't know about the new file we haven't updated it yet. We should land that so that reality and ansible are in sync. I will approve it now | 19:46 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 978956: Expire mod_security collection entries in one day https://review.opendev.org/c/opendev/system-config/+/978956 | 20:07 | |
| @fungicide:matrix.org | load on static02 has spiked pretty mightily after the deploy, but seems to be calming down slowly | 20:14 |
| @fungicide:matrix.org | i'm still having trouble loading pages, though server-status says it's not due to busy worker slots | 20:15 |
| @fungicide:matrix.org | when i first looked the 15-minute load was over 25, the 5-minute load is now down below 8 | 20:15 |
| @fungicide:matrix.org | and it's continuing to fall | 20:16 |
| @fungicide:matrix.org | i do wonder if the vhost config change invalidated the honeypot table and it's rebuilding from scratch as new requests come in for the bad paths now | 20:17 |
| @fungicide:matrix.org | a page i was previously stuck trying to load just popped in when i reloaded in my browser, so i think it's more or less recovered now | 20:18 |
| @clarkb:matrix.org | Or maybe the apache restart did it? | 20:18 |
| @fungicide:matrix.org | possible | 20:18 |
| @clarkb:matrix.org | I think the other changes were all reloads but this last one was a restart | 20:19 |
| @clarkb:matrix.org | Though it should've only triggered if the file changed? | 20:19 |
| @clarkb:matrix.org | That may be worth checking in the service-static log on bridge | 20:19 |
| @fungicide:matrix.org | the replacement file may not have been byte-for-byte identical | 20:19 |
| @clarkb:matrix.org | Good point | 20:19 |
| @fungicide:matrix.org | though /etc/modsecurity/collection-timeout.conf was last updated on 2026-03-04 looks like | 20:21 |
| @clarkb:matrix.org | so maybe it was the vhost reloads instead since those were definitely updated | 20:24 |
| @clarkb:matrix.org | yes the Reload apache handler ran at 19:36:13 according to the most recent log | 20:26 |
| @clarkb:matrix.org | which is actually earlier than the deployment for the collection-teimout.conf file | 20:26 |
| @clarkb:matrix.org | huh the playbook that recorded 19:36 started at 20:10 so maybe we have a bad clock somewhere or I'm looking at the wrong timestamp | 20:27 |
| @clarkb:matrix.org | anyway I think it did reload but not restart with the last deployment and maybe that was enough to do it | 20:27 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!