| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 1000797: Add mnasiadka to accessbot admins https://review.opendev.org/c/openstack/project-config/+/1000797 | 06:10 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [openstack/project-config] 1000797: Add mnasiadka to accessbot admins https://review.opendev.org/c/openstack/project-config/+/1000797 | 06:28 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 998419: Add check-sources propose job for kolla https://review.opendev.org/c/openstack/project-config/+/998419 | 07:02 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 998419: Add check-sources propose job for kolla https://review.opendev.org/c/openstack/project-config/+/998419 | 07:05 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 999465: Compact borg repos after pruning https://review.opendev.org/c/opendev/system-config/+/999465 | 08:08 | |
| @gthiemonge:matrix.org | Hey Folks, we're facing a lot of "ModuleNotFoundError: No module named 'openstack_requirements.cmds'" issues in the CI | 13:52 |
|---|---|---|
| https://zuul.opendev.org/t/openstack/build/1c7bb96d4e4d48a38088aa560a09e821 | ||
| Any ideas? | ||
| @fungicide:matrix.org | gtema: try again, i yanked the pbr release from a few hours ago, fix is already being reviewed | 14:00 |
| @fungicide:matrix.org | er, gthiemonge ^ (sorry gtema, overzealous enter after incorrect tab completion) | 14:00 |
| @gthiemonge:matrix.org | fungi: ack, thanks, I'll try | 14:06 |
| @clarkb:matrix.org | mnasiadka: any idea on why the node exporter stuff doesn't seem to have started? | 15:14 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1000864: Bind mount dir for haproxy log socket not the socket itself https://review.opendev.org/c/opendev/system-config/+/1000864 | 15:57 | |
| @mnasiadka:matrix.org | Clark: not yet, but will have a look in some minutes | 15:58 |
| @mnasiadka:matrix.org | Clark: I can see prometheus node exporter running on prometheus01 and I can get the metrics (curl -L localhost:9090/api/v1/label/__name__/values | python3 -m json.tool) | 16:07 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1000864: Bind mount dir for haproxy log socket not the socket itself https://review.opendev.org/c/opendev/system-config/+/1000864 | 16:08 | |
| @clarkb:matrix.org | mnasiadka: https://prometheus.opendev.org/targets is green now | 16:08 |
| @clarkb:matrix.org | mnasiadka: so maybe the daily run fixed things? May need to look at the logs for the deployment run on bridge to see if there is anything obviously wrong | 16:09 |
| @clarkb:matrix.org | I would've expected the job to fail though if something like a container image pull failed | 16:09 |
| @mnasiadka:matrix.org | Clark: yes, daily run had to fix it - the patch I raised didn't touch anything prometheus wise in patch to enable the deploy jobs, so it didn't run in deploy pipeline | 16:09 |
| @mnasiadka:matrix.org | Clark: Do we need to guard Prometheus with some authentication, or public access to these metrics is fine? | 16:10 |
| @clarkb:matrix.org | mnasiadka: historically we've allowed public access to this info. But that is why in previous reviews on the stack I suggested we would need to audit the data that is being collected | 16:11 |
| @clarkb:matrix.org | I think it is possible there is some data we need to exclude from collection so that we can share everything else otherwise | 16:11 |
| @clarkb:matrix.org | and yes I see now that infra-prod-node-exporter did not run when you added the job | 16:12 |
| @clarkb:matrix.org | so that explains it. Had to wait for the hourly run | 16:12 |
| @clarkb:matrix.org | re auditing I think we are now in a positiion to start doing that with the actual data rather than theories. So the next step is probably to review what we are collecting. Remove anything we're concerned about and then proceed from there | 16:13 |
| @jim:acmegating.com | https://prometheus.opendev.org/api/v1/targets/metadata may be helpful | 16:14 |
| @jim:acmegating.com | and https://prometheus.opendev.org/query?g0.expr={__name__%3D~%22node_.%2B%22} | 16:17 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [openstack/project-config] 1000868: Update openstack-github-mirroring API token https://review.opendev.org/c/openstack/project-config/+/1000868 | 16:23 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 1000873: prometheus: Add scraping of greptimedb metrics https://review.opendev.org/c/opendev/system-config/+/1000873 | 16:51 | |
| @mnasiadka:matrix.org | Clark: ^^ that might be useful | 17:53 |
| @clarkb:matrix.org | yup I think gathering info in the current state that is representative but low risk due to minimal scope is a good idea | 17:54 |
| @clarkb:matrix.org | that will allow us to evaluate and tune before we throw all the things at it | 17:54 |
| @clarkb:matrix.org | I'm going to try and run some errands around lunch today, but https://review.opendev.org/c/opendev/system-config/+/1000347 is one that may interest you fungi (should correct the health checks for anubis on lists.o.o. If you do a `docker ps -a` there you'll see it says anubis is unhealthy and that is due to being unable to reach the metrics endpoint to check properly | 18:23 |
| @fungicide:matrix.org | i approved 1000347, i'm going to grab dinner and run a quick errand but shouldn't be long | 19:24 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1000347: Correct lists anubis metrics port config https://review.opendev.org/c/opendev/system-config/+/1000347 | 19:59 | |
| @clarkb:matrix.org | cool anubis says it is healthy now on lists | 21:10 |
| @fungicide:matrix.org | so much the better | 21:11 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1000909: Upgrade gitea to 1.27.2 https://review.opendev.org/c/opendev/system-config/+/1000909 | 21:23 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed wip: [openstack/project-config] 1000918: Replace 2026.2/Hibiscus key with 2027.1/Indri https://review.opendev.org/c/openstack/project-config/+/1000918 | 22:37 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!