| -@gerrit:opendev.org- Anil Shashikumar Belur proposed on behalf of Clark Boylan: [opendev/system-config] 983318: Use Haproxy PROXY Protocol with Gitea https://review.opendev.org/c/opendev/system-config/+/983318 | 04:11 | |
| -@gerrit:opendev.org- Anil Shashikumar Belur proposed on behalf of Clark Boylan: [opendev/system-config] 983318: Use Haproxy PROXY Protocol with Gitea https://review.opendev.org/c/opendev/system-config/+/983318 | 11:40 | |
| @fungicide:matrix.org | looking at https://grafana.opendev.org/d/9871b26303/afs i'm going to increase the mirror.ubuntu-ports quota from 1.1tb to 1.2tb | 15:39 |
|---|---|---|
| @fungicide:matrix.org | also mirror.epel and mirror.fedora are stale (we should just stop running the fedora rsync anyway, it's empty now) | 15:39 |
| @fungicide:matrix.org | looks like the epel mirroring problem is that pubmirror3.math.uh.edu is timing out the rsync socket connection | 15:45 |
| @fungicide:matrix.org | https://review.opendev.org/c/opendev/system-config/+/862793 (4 years ago) Dmitriy Rabotyagov switched it to that from pubmirror1.math.uh.edu after conferring with the mirrors' maintainer | 15:47 |
| @fungicide:matrix.org | comments in the file indicate paul had long ago conferred with tibbs in the #fedora-admin irc channel about the pubmirror*.math.uh.edu rsync services | 15:49 |
| @fungicide:matrix.org | i've reached out there, and in the meantime am testing rsync from pubmirror2 which seems to have current content | 15:57 |
| @fungicide:matrix.org | #status log Increased AFS quota for the mirror.ubuntu-ports volume by 100GB (from 1.1TB to 1.2TB) in order to make a little more room for growth | 16:24 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 16:25 |
| @fungicide:matrix.org | though with pubmirror2 i was getting checksum mismatches against https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/repodata/repomd.xml so i ended up mirroring from publirror1 finally which seemed to be more current/correct | 16:39 |
| @fungicide:matrix.org | still waiting to hear back before i push any changes for the script though | 16:40 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 998978: Use matrix ids in matrix meeting logs https://review.opendev.org/c/opendev/system-config/+/998978 | 16:56 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 998979: Add meetings/ to matrix-eavesdrop meetings URL https://review.opendev.org/c/opendev/system-config/+/998979 | 16:58 | |
| @fungicide:matrix.org | aha, thanks! | 16:58 |
| @jim:acmegating.com | fungi: Clark Anil Belur ^ two things noticed in the testing fungi and i just did | 16:58 |
| @fungicide:matrix.org | it's getting very close | 16:58 |
| @jim:acmegating.com | ++ | 16:59 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 17:50 | |
| - [opendev/system-config] 998978: Use matrix ids in matrix meeting logs https://review.opendev.org/c/opendev/system-config/+/998978 | ||
| - [opendev/system-config] 998979: Add meetings/ to matrix-eavesdrop meetings URL https://review.opendev.org/c/opendev/system-config/+/998979 | ||
| @fungicide:matrix.org | grr, apache is swamped on the lists01 server, even with anubis in place. not responding at all for m | 18:23 |
| @fungicide:matrix.org | e | 18:23 |
| @fungicide:matrix.org | it finally recovered, i think whatever bot army was pounding it moved on for now | 18:44 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 19:21 | |
| - [opendev/system-config] 998978: Use matrix ids in matrix meeting logs https://review.opendev.org/c/opendev/system-config/+/998978 | ||
| - [opendev/system-config] 998979: Add meetings/ to matrix-eavesdrop meetings URL https://review.opendev.org/c/opendev/system-config/+/998979 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 995420: Drop backup02 from our inventory https://review.opendev.org/c/opendev/system-config/+/995420 | 19:25 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 848181: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 19:40 | |
| @clarkb:matrix.org | I put an autohold in place for ^ to directly check a gitea 1.27.1 install | 19:41 |
| @clarkb:matrix.org | corvus: Anil Belur 998978 lgtm but it fails on a test assertion due to the change of the id values (so just need to update the test cases I made a note on the change) | 19:48 |
| @jim:acmegating.com | Clark: i fixed that maybe right before your comment | 19:48 |
| @clarkb:matrix.org | oh so you did +1 from me | 19:49 |
| @clarkb:matrix.org | * oh so you did +2 from me | 19:49 |
| @clarkb:matrix.org | did gerritbot not notify of that in here? | 19:49 |
| @clarkb:matrix.org | no it did it just took a bit to run CI jobs | 19:50 |
| @clarkb:matrix.org | I'm caught up now. Sorry for the noise | 19:50 |
| @jim:acmegating.com | \o/ | 19:50 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 998978: Use matrix ids in matrix meeting logs https://review.opendev.org/c/opendev/system-config/+/998978 | 20:44 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [openstack/project-config] 999019: Update GitHub mirroring credentials for OpenStack https://review.opendev.org/c/openstack/project-config/+/999019 | 20:46 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 998979: Add meetings/ to matrix-eavesdrop meetings URL https://review.opendev.org/c/opendev/system-config/+/998979 | 20:55 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 999021: Drop Fedora mirroring https://review.opendev.org/c/opendev/system-config/+/999021 | 20:58 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: | 21:06 | |
| - [openstack/project-config] 999019: Update GitHub mirroring credentials for OpenStack https://review.opendev.org/c/openstack/project-config/+/999019 | ||
| - [openstack/project-config] 999022: Don't print GitHub auth responses as exceptions https://review.opendev.org/c/openstack/project-config/+/999022 | ||
| @fungicide:matrix.org | infra-prod-service-grafana failed deploying the backup02 removal | 21:13 |
| @clarkb:matrix.org | I don't think we backup that server as all of its data is in config management so probably unrelated to the change? | 21:13 |
| @fungicide:matrix.org | `PUT /api/datasources/1` returned `requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://admin:...@localhost:3000/api/datasources/1` | 21:14 |
| @fungicide:matrix.org | (the actual exception contains the admin pw, so omitted above) | 21:15 |
| @fungicide:matrix.org | looks like that job started failing 2026-06-11 in periodic (last successful run was 2026-06-10 in deploy) | 21:21 |
| @fungicide:matrix.org | which incidentally was for the `Update grafana to 13.0.2` change | 21:22 |
| @fungicide:matrix.org | so i expect something about the upgrade broke that api | 21:22 |
| @fungicide:matrix.org | we upgraded from 12.0.2 to 13.0.2, probably a v13 backward-incompatible change | 21:23 |
| @fungicide:matrix.org | https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v13-0/#breaking-changes | 21:24 |
| @fungicide:matrix.org | "Deprecated data source APIs disabled by default: Data source APIs that referenced data sources using a numeric id were deprecated in Grafana v9.0 in favor of APIs that reference data sources using a uid instead. In Grafana v13.0, these deprecated APIs are disabled by default." | 21:25 |
| @fungicide:matrix.org | https://grafana.com/whats-new/2026-04-14-deprecated-data-source-apis-disabled-by-default/ | 21:26 |
| @fungicide:matrix.org | looks like we don't currently have a grafana.ini or custom.ini that i can find, so it may be easier to update to use uuids | 21:28 |
| @clarkb:matrix.org | Weird that the 13.0.2 change passed ci | 21:29 |
| @fungicide:matrix.org | yeah, it even deployed, but then started failing once deployed | 21:30 |
| @clarkb:matrix.org | 174.143.59.81 is the held gitea load balancer. You have to put that in /etc/hosts for opendev.org due to the anubis domain checks. But it seems to work if I browse opendev/system-config on that server | 21:36 |
| @clarkb:matrix.org | fungi: I think those errors come from grafyaml | 21:38 |
| @clarkb:matrix.org | I wonder if it automatically uses uuids in a new install and so things work, but with our existing install things have int ids instead and then it fails | 21:39 |
| @clarkb:matrix.org | yes grafyaml creates sha256sum uid values for newly created datasources. I'm thinking that this may be an issue with our preexisting data? | 21:40 |
| @clarkb:matrix.org | https://opendev.org/opendev/grafyaml/src/branch/master/grafana_dashboards/grafana/datasource.py#L76-L77 | 21:41 |
| @clarkb:matrix.org | I wonder if we just have it start over with a new database if it would just work. Of course the risk there is that if it doesn't work we have no graphs anymore | 21:42 |
| @clarkb:matrix.org | mnasiadka: I think the node exporter and greptimedb changes lgtm | 22:06 |
| @clarkb:matrix.org | fungi: hrm by default grafana stores its db in a sqlite file and I don't think we're bind mounting that file so my earlier theory is probably wrong | 22:11 |
| @clarkb:matrix.org | fungi: ok creating a dashboard and updating a dashboard use different paths in grafyaml. I believe this is the issue | 22:17 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 22:25 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 999031: Run grafyaml twice in testing https://review.opendev.org/c/opendev/system-config/+/999031 | 22:28 | |
| @clarkb:matrix.org | that system-config change should fail as is. Once we have confirmed that reproduces the issue we can depends on 999030 and check that the problem is fixed. I made a note of this in the commit message for 999031 | 22:28 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 22:31 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 22:38 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 999031: Run grafyaml twice in testing https://review.opendev.org/c/opendev/system-config/+/999031 | 23:07 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 23:44 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 23:57 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!