| -@gerrit:opendev.org- Clark Boylan proposed: | 16:02 | |
| - [opendev/system-config] 988279: Mirror Ubuntu Resolute Docker packages https://review.opendev.org/c/opendev/system-config/+/988279 | ||
| - [opendev/system-config] 988280: Mirror Ubuntu Resolute Packages https://review.opendev.org/c/opendev/system-config/+/988280 | ||
| @clarkb:matrix.org | That is mostly for discussion and helping to move things forward. I haven't done much determination of whether or not this will fit into the existing afs volume | 16:02 |
|---|---|---|
| @fungicide:matrix.org | i manually downloaded the truncated packages into the mirror.ubuntu-ports volume and am rerunning reprepro for it now | 16:43 |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 17:47 | |
| @clarkb:matrix.org | oh cool /me immediately adds reviewing that change to the todo list | 17:50 |
| @fungicide:matrix.org | ooh | 17:51 |
| @fungicide:matrix.org | that's https://github.com/GrepTimeTeam/greptimedb i guess | 17:53 |
| @fungicide:matrix.org | "Up to 50x lower storage" | 17:53 |
| @fungicide:matrix.org | i guess it's the "up to" part that will need some investigating | 17:54 |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 18:06 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 18:07 | |
| @mnasiadka:matrix.org | Clark: I think I would need to understand how do we manage secrets in system-config - because in current form greptimedb has no authentication - but we could easily do HTTP basic auth on greptimedb | 18:10 |
| @fungicide:matrix.org | for prometheus-to-greptime communication, or is this user-to-greptime access? | 18:12 |
| @mnasiadka:matrix.org | Well, without any auth - anybody can push data in there (and anybody can read it, but that's not that problematic) | 18:12 |
| @mnasiadka:matrix.org | * Well, without any auth - anybody can push data in there (and anybody can read it, but that's not that problematic given the data is public) | 18:12 |
| @mnasiadka:matrix.org | So we could just do authentication on /v1/prometheus/write Location | 18:13 |
| @mnasiadka:matrix.org | And that would be only for Prometheus-to-GrepTimeDB pushes | 18:14 |
| @fungicide:matrix.org | right, but if it's just a backend for prometheus, we could set iptables/netfilter to only allow connections from the prometheus server | 18:15 |
| @mnasiadka:matrix.org | yes, that's true | 18:15 |
| @fungicide:matrix.org | we do that in other places | 18:17 |
| @mnasiadka:matrix.org | Ok, let me have a look and replicate that approach | 18:17 |
| @fungicide:matrix.org | for example, statsd includes no authentication | 18:17 |
| @mnasiadka:matrix.org | Second question probably is do we need HTTPS for GrepTimeDB - and with that approach we probably don't. | 18:18 |
| @fungicide:matrix.org | if it's not user-facing, then likely not necessary if it complicates things | 18:18 |
| @clarkb:matrix.org | ya I think the assumption was that prometheus would collect data from nodes. That would be controlled via firewall rules. Then prometheus can write to somewhere else (possibly with auth if necessary)? | 18:19 |
| @mnasiadka:matrix.org | The architecture is sort of this: | 18:21 |
| User facing: Grafana -> Prometheus (HTTPS) -> local TSDB (short-term) -> GrepTimeDB port 4000 (long-term, iptables-restricted, HTTP) | ||
| Metrics scraping: Prometheus connects to node_exporters of all scraped servers and saves data in local TSDB (short-term) and then data lands in GrepTimeDB (long-term) | ||
| @mnasiadka:matrix.org | So I think HTTP only with iptables limited connectivity only from Prometheus server is simplistic approach | 18:22 |
| @mnasiadka:matrix.org | (and we don't need Apache on GrepTimeDB then) | 18:22 |
| @clarkb:matrix.org | that seems reasonable as a starting point | 18:25 |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 18:26 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 18:36 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 19:04 | |
| @fungicide:matrix.org | the mirror.ubuntu-ports volume is showing caught up and released now according to grafana | 19:05 |
| @fungicide:matrix.org | though now mirror.debian-security looks like it's gotten into the same situation 3 days ago, so i'll look at that next | 19:05 |
| @fungicide:matrix.org | #status log Increased the mirror.debian-security AFS volume quota by 50GB (from 350GB to 400GB) bringing utilization down to 88% | 19:09 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 19:09 |
| @fungicide:matrix.org | i'm manually running reprepro for that in a root screen session on mirror-update now | 19:10 |
| @fungicide:matrix.org | will see what needs repairing, hopefully not much | 19:11 |
| @fungicide:matrix.org | looks like there are a few more truncated files this time. 5 that need fixing i think | 19:22 |
| @fungicide:matrix.org | okay, truncated packages have been manually replaced with official copies, and reprepro is running again, hopefully to completion this time | 19:31 |
| -@gerrit:opendev.org- Jack Hodgkiss proposed: [openstack/diskimage-builder] 986427: fix: add support for `cloud-init` in `Ubuntu Resolute` https://review.opendev.org/c/openstack/diskimage-builder/+/986427 | 19:36 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 19:52 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Takashi Kajinami: [openstack/diskimage-builder] 987986: Suppress warning from eventlet https://review.opendev.org/c/openstack/diskimage-builder/+/987986 | 20:29 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 20:43 | |
| -@gerrit:opendev.org- Michal Nasiadka marked as active: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 20:43 | |
| @fungicide:matrix.org | the mirror.debian-security volume is showing recently released in grafana now, so afs storage estimates should be accurate at this point | 20:53 |
| @fungicide:matrix.org | 104.239.175.4 seems to be the held etherpad 2.7.3 node, and i can interact normally with the test pad, so i've gone ahead and approved 985843 | 20:54 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 985843: Upgrade etherpad to 2.7.3 https://review.opendev.org/c/opendev/system-config/+/985843 | 21:23 | |
| @clarkb:matrix.org | looks like that deployment job is going to fail for some reason | 21:28 |
| @clarkb:matrix.org | it hit docker rate limits | 21:30 |
| @clarkb:matrix.org | I can manually edit /etc/hosts to use ipv4 addresses for docker things and reenqueue the buildset in a bit | 21:30 |
| @fungicide:matrix.org | oh, right, ipv6 aggregation resulting in basically all of rackspace getting treated as a single address by dockerhub's rate-limiter | 21:34 |
| @clarkb:matrix.org | I've updated /etc/hosts with ipv4 overrides and will reenqueue the buildset now | 21:39 |
| @clarkb:matrix.org | apparently promoting container images is not idempotent | 21:41 |
| @clarkb:matrix.org | the promotion job for the etherpad image has failed | 21:41 |
| @clarkb:matrix.org | I guess because the old image tag gets removed when we mvoe it to latest? | 21:41 |
| @clarkb:matrix.org | I will manually pull and down then up -d so that we don't wait for the daily runs to do that | 21:43 |
| @clarkb:matrix.org | that is done. I think the service is up and running | 21:50 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/gerrit-upgrade-3.13 renders for me as well | 21:50 |
| @clarkb:matrix.org | and /etc/hosts has been restored to its prior state | 21:52 |
| @clarkb:matrix.org | mnasiadka: ok posted some thoughts to https://review.opendev.org/c/opendev/system-config/+/988310/ In particular I wonder if we can run prometheus and greptimedb on the same host and simplify things a bit? | 23:10 |
| @clarkb:matrix.org | let me know what you think | 23:10 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!