| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/995777 | 04:28 |
|---|---|---|
| noonedeadpunk | So https://review.opendev.org/c/openstack/openstack-ansible/+/995111 seems to be working really well | 07:33 |
| noonedeadpunk | I have additionally tested it with https://review.opendev.org/c/openstack/openstack-ansible-os_freezer/+/983870 where I change tempest plugin, and I see behavior changes in tempest for surte | 07:33 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/995777 | 07:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_freezer master: Remove v1 API https://review.opendev.org/c/openstack/openstack-ansible-os_freezer/+/991716 | 08:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Add boilerplate for observability collection https://review.opendev.org/c/openstack/openstack-ansible-ops/+/995787 | 08:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Add boilerplate for observability collection https://review.opendev.org/c/openstack/openstack-ansible-ops/+/995787 | 09:01 |
| mgariepy | noonedeadpunk: do we merge the Depends-On fix then ? | 10:29 |
| noonedeadpunk | mgariepy: we need https://review.opendev.org/c/openstack/openstack-ansible/+/995102/2 first :) | 10:30 |
| noonedeadpunk | or well | 10:30 |
| noonedeadpunk | we can rebase as it's not "required" | 10:30 |
| noonedeadpunk | but it was hard for me to see the content without data being collected | 10:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Respect Depends-On for service projects https://review.opendev.org/c/openstack/openstack-ansible/+/995111 | 10:30 |
| noonedeadpunk | I just realized we can decouple these | 10:30 |
| mgariepy | okie. | 10:32 |
| mgariepy | victorialogs still holding strong. | 10:32 |
| noonedeadpunk | nice | 10:32 |
| noonedeadpunk | I'm just now got into looking on adding observability as a collection | 10:32 |
| mgariepy | 4.35B, ~240gb on disk. | 10:32 |
| noonedeadpunk | really nice result | 10:32 |
| mgariepy | full text search on last 30 days is like 15 sec result. | 10:33 |
| noonedeadpunk | ok, that's what I was worried about | 10:33 |
| mgariepy | all this on a 8 core 32gb vm. | 10:33 |
| mgariepy | the way it works is really clever ;) | 10:34 |
| noonedeadpunk | it sounds like a very nice alternative to opensearch then | 10:34 |
| mgariepy | and you also do need to pre-hash all your log. | 10:34 |
| mgariepy | do not** | 10:34 |
| noonedeadpunk | yeah, pre-hashing is unrealistic in world with dozen different sources | 10:35 |
| mgariepy | it's really impressive. | 10:35 |
| mgariepy | i do injest between 2M and 10M logs per hours. | 10:36 |
| mgariepy | if you need to extract some data from your query, it's also very easy. | 10:37 |
| mgariepy | and you can search on any field, not just limited to a few fields. | 10:39 |
| noonedeadpunk | that's what fulltext mean?:) | 10:39 |
| mgariepy | about 9.6TB down to ~240GB | 10:40 |
| mgariepy | i wonder what else is out there that we don't know about:) haha | 10:40 |
| mgariepy | our org might be ditching opensearch soonish. | 10:41 |
| mgariepy | openstack + ceph is the most chatty cluster tho lol. | 10:42 |
| noonedeadpunk | Oh, yes, sure | 10:44 |
| mgariepy | about 330 host does generate a lot of data :) | 10:44 |
| noonedeadpunk | Well, switches can be also quite chatty I'd say | 10:44 |
| noonedeadpunk | but nothing compared to these 2 | 10:44 |
| mgariepy | and i know i'm missing some logs | 10:44 |
| mgariepy | anyway. it's kinda nice to be able to track what happens to a single file in object store since a certain date :D | 10:46 |
| mgariepy | ho. 380 hosts, 330 was only the last 5 minutes haha | 10:47 |
| noonedeadpunk | have you also played with alerting on logs there? | 10:48 |
| noonedeadpunk | liek rate of errors from a service and things like that? | 10:48 |
| mgariepy | not yet, but i started building some grafana dashboard | 10:48 |
| mgariepy | i do have a small graph for traceback count per service. | 10:49 |
| mgariepy | but my goal is to have all this at some point. but it might be more like a medium goal. | 10:50 |
| mgariepy | a nice thing is that we can have dashboard for both logs and metrics at the same place. | 10:51 |
| noonedeadpunk | I'd argue you can do that with opensearch as well? | 10:53 |
| mgariepy | our opensearch cluster wasn't able to follow up my logs much. | 10:54 |
| mgariepy | and the pre-hashing thing was an issue. | 10:54 |
| noonedeadpunk | yes, right, scaling is bigger problem indeed | 10:54 |
| mgariepy | without my cloud in it, i think when the openstack cluster is down for maintenance it takes a few hours to sync back the logs. | 10:55 |
| mgariepy | few here is 12h+ | 10:56 |
| mgariepy | and maintenance is also a log longer than a single server for victorialogs :) haha | 10:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 17:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 17:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 18:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 18:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 18:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 18:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_freezer master: Add support for centralized scheduler deployment https://review.opendev.org/c/openstack/openstack-ansible-os_freezer/+/983870 | 18:52 |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server master: Set RABBITMQ_MAX_NUMBER_OF_PROCESSES variable https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/995883 | 19:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 19:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Grafana: fix db creation, remove built-in plugin, refresh and introduce install docs https://review.opendev.org/c/openstack/openstack-ansible-ops/+/984878 | 20:29 |
| opendevreview | Dmitriy Chubinidze proposed openstack/openstack-ansible-openstack_hosts master: Add tar and unzip to distro packages https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/995898 | 20:47 |
| opendevreview | Dmitriy Chubinidze proposed openstack/openstack-ansible-openstack_hosts master: Add tar and unzip to distro packages https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/995898 | 20:48 |
| *** zseguin_ is now known as zseguin | 23:53 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!