| gokhan | Hello kolks, container logs are under /openstack/log/ dir. There also journald logs of services. How do you manage rotation of these logs. For systemd-journald logs it seems we need to set configs in all of the containers. https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks/logging.html | 06:49 |
|---|---|---|
| gokhan | *folks | 06:49 |
| noonedeadpunk | gokhan: you actually don't have to | 06:59 |
| noonedeadpunk | there are "reasonable" defaults on side of systemd-journald, which handles log rotation | 06:59 |
| noonedeadpunk | regarding /openstack/log/ - I'm not sure if it's much of a use these days | 07:00 |
| noonedeadpunk | check this out: https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html#SystemMaxUse= | 07:00 |
| noonedeadpunk | so pretty much by default journald is capped at 4gb or smth | 07:01 |
| noonedeadpunk | but you can tune it in a really flexible ways | 07:01 |
| noonedeadpunk | jrosser: damiandabrowski I kinda also would be using ternary instead of if/else but I assumed it's just what I got used to, so was not commenting in this regard. As functionally it's the same. | 07:03 |
| noonedeadpunk | unless talking about full-blown Jinja, where you can do some formatting | 07:04 |
| jrosser | I don’t really understand the point of the reflex either there | 07:04 |
| jrosser | regex | 07:04 |
| jrosser | I do I think for the backward compatibility case, but less so for the commented line which I assume to be what we put in when we only support ttl | 07:05 |
| gokhan | noonedeadpunk, it is about 342 gb /openstack/log https://paste.openstack.org/show/bFVA0cOxhhGKLyJnM7Oi/. Especially magnum and galera. we are using cluster api at magnum side | 07:06 |
| jrosser | if there is a reason there should maybe be some explanation as the regex is unreadable to me basically | 07:06 |
| noonedeadpunk | jrosser: regex was my doing I believe :D | 07:15 |
| noonedeadpunk | but prior way was string splitting, which I find a terrible idea, as there could be a wide variety of input uncluding spaces, tabs and whatsoever in between | 07:16 |
| noonedeadpunk | gokhan: ugh | 07:17 |
| jrosser | ok well maybe the commit message the. just sends me to bit understanding it so all | 07:17 |
| jrosser | argh | 07:17 |
| jrosser | I basically am finding that patch impossible to review | 07:17 |
| noonedeadpunk | regexp specifically does this in a different way: https://review.opendev.org/c/openstack/ansible-role-pki/+/948880/8/tasks/standalone/create_ca.yml | 07:20 |
| noonedeadpunk | and yes, that I assume is for like backwards compatability | 07:21 |
| noonedeadpunk | gokhan: huh... interesting... what what is actually producing these 170G inside of the container? any specific log file? | 07:22 |
| gokhan | noonedeadpunk, for magnum it is running in debug mode, for galera there are mariadb check logs in syslog. syslog is producing | 07:27 |
| noonedeadpunk | even though it's running in debug mode, it should be logging to journald right? | 07:32 |
| noonedeadpunk | syslog should actually be rotated by the system as well afaik... but I obviously more concerned about reasons why magnum takes so much of diskspace | 07:35 |
| noonedeadpunk | jrosser: let me see if I can improve readability of this | 07:53 |
| noonedeadpunk | I personally was not voting as I'm actually on the fence with the idea itself | 07:53 |
| noonedeadpunk | It's kinda fair that formats are different, and indeed ttl might make more sense... | 07:54 |
| jrosser | is this all really just .startswith() | 07:54 |
| noonedeadpunk | but like we're chaning well established driver with quite some complexity to be in line with new driver approach | 07:54 |
| noonedeadpunk | yeah, kinda, except accounting for spaces/tabs in the beginning | 07:55 |
| noonedeadpunk | ie https://regex101.com/r/r8sgEt/1 | 07:58 |
| noonedeadpunk | if match - do nothing, if does not match - add + at the beginning | 07:59 |
| jrosser | oh my | 08:01 |
| damiandabrowski | jrosser: the idea was to add "+" at the beginning when its not there - to make `selfsigned_not_after` work together with both `ttl` and `not_after` format, for a one release | 08:01 |
| damiandabrowski | originally it worked like this: | 08:01 |
| damiandabrowski | selfsigned_not_after: "{{ ('+' if not (ca.not_after | default(ca.ttl))[:1] == '+' else '') + (ca.not_after | default(ca.ttl)) }}" | 08:02 |
| damiandabrowski | but noonedeadpunk suggested regex | 08:02 |
| noonedeadpunk | Yeah, I mentioned that it was my take :p | 08:02 |
| jrosser | can you then explain to me the commented out line (122) | 08:02 |
| jrosser | which i understand to be the line we will use after removing backward compat | 08:03 |
| damiandabrowski | yeah, but now I wonder if it's correct, because after dropping backwards compatbility we probably won't need to have any logic there and always add leading + | 08:06 |
| damiandabrowski | selfsigned_not_after: "+{{ ca.ttl }}" | 08:06 |
| damiandabrowski | ^ so something like this may be enough | 08:06 |
| noonedeadpunk | yeah, I think it would make sense to do like that when we drop compatability | 08:09 |
| noonedeadpunk | but we need to have some kind of upgrade check for dropping it | 08:09 |
| noonedeadpunk | as CA are being redefined for pretty much all production deployments, and it's super easy to forget to update them | 08:09 |
| noonedeadpunk | so I'm not really sure how to drop this compatability nicely | 08:10 |
| jrosser | perhaps we need a filter rather than doing this in native ansible | 08:18 |
| jrosser | then it can be readable code with comments and error handling blah blah | 08:18 |
| jrosser | `(ca.not_after | default(ca.ttl)) | normalise_ttl()` | 08:19 |
| gokhan | noonedeadpunk, because of log rotation on magnum journald log size is only 3g. we also need log rotation on syslog | 08:41 |
| noonedeadpunk | gokhan: what is being written to syslog though? | 08:44 |
| noonedeadpunk | also are you running ubuntu? | 08:44 |
| noonedeadpunk | I think there should be native logrotate configuration for it tbh... | 08:45 |
| noonedeadpunk | ie `/etc/logrotate.d/rsyslog` or smth | 08:46 |
| gokhan | I am using ubuntu | 08:47 |
| gokhan | https://paste.openstack.org/show/bnFPjLMBodwXtiLVBJUD/ | 08:48 |
| gokhan | we need to add also for lxc containers | 08:48 |
| noonedeadpunk | I wonder if `ForwardToSyslog=no` should be just set | 08:49 |
| noonedeadpunk | in `openstack_hosts_journald_config` | 08:49 |
| noonedeadpunk | not to duplicate journald to the syslog | 08:50 |
| jrosser | ^ we did this here recently on all our hosts | 08:51 |
| noonedeadpunk | but I'm not sure it's the case inside of LXC tbh | 08:51 |
| noonedeadpunk | I'm not having /var/log/syslog on some modern enough deployemtn | 08:52 |
| noonedeadpunk | https://paste.openstack.org/show/bp7xQMOdGDPcVCejb4v4/ | 08:53 |
| noonedeadpunk | but it's ubuntu 24.04 | 08:54 |
| noonedeadpunk | so probably that is already solved in a way :D | 08:55 |
| noonedeadpunk | but it feels reasonable to add this parameter to documentation | 08:55 |
| gokhan | noonedeadpunk, I am using ubuntu 22.04 and magnum send logs to syslog https://paste.openstack.org/show/bdIikjPbzP6i1CqPKRVH/ | 09:07 |
| gokhan | https://paste.openstack.org/show/bCOcr7cRry6exj9rcKyn/ | 09:08 |
| noonedeadpunk | yeah, this should be doing the trick I guess | 09:23 |
| noonedeadpunk | or well. at least prevent syslog from growing again | 09:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Switch OpenStack codename for 2025.2 https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/962600 | 13:12 |
| opendevreview | Merged openstack/openstack-ansible master: Ensure releasing does not inject new lines https://review.opendev.org/c/openstack/openstack-ansible/+/962333 | 14:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Switch OpenStack codename for 2025.2 https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/962600 | 14:07 |
| *** Guest28048 is now known as jotik | 17:33 | |
| opendevreview | Damian Dąbrowski proposed openstack/ansible-role-pki master: Use ttl instead of not_after in pki_authorities https://review.opendev.org/c/openstack/ansible-role-pki/+/948880 | 19:01 |
| opendevreview | Damian Dąbrowski proposed openstack/ansible-role-pki master: Add hashi_vault backend https://review.opendev.org/c/openstack/ansible-role-pki/+/948881 | 19:01 |
| opendevreview | Damian Dąbrowski proposed openstack/ansible-role-pki master: Add hashi_vault backend https://review.opendev.org/c/openstack/ansible-role-pki/+/948881 | 19:02 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!