| opendevreview | Matt Crees proposed openstack/kayobe stable/2023.1: Add script to migrate to RabbitMQ quorum queues https://review.opendev.org/c/openstack/kayobe/+/919925 | 08:31 |
|---|---|---|
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: rework RMQ steps for queue migrations https://review.opendev.org/c/openstack/kayobe/+/913878 | 08:33 |
| kevko | mnasiadka: This is everything what is in LOG for keepalived l3 agent ... https://paste.openstack.org/show/bq0Ixf8HNqshvWU7Jpap/ i was stopping other l3 agents ...nothing logged .. what i am missing ? | 09:22 |
| kevko | (compiled keepalived with log-file support) | 09:22 |
| SvenKieske | can someone RP+1 this? or possibly just +2 it, imho a patch we should asap backport (I can do it): https://review.opendev.org/c/openstack/kolla-ansible/+/919490 | 09:23 |
| SvenKieske | kevko: I'm still confused, is there some bugreport somewhere describing the issue in detail? | 09:24 |
| kevko | SvenKieske: what do you mean ? | 09:38 |
| kevko | SvenKieske: approved btw ^^ | 09:38 |
| kevko | SvenKieske: I remember that lastly i created service role manually :P | 09:38 |
| kevko | so nice to see that patch | 09:38 |
| SvenKieske | you posted about problem with keepalived logging and I don't really understand the problem, is there a bug in keepalived logging? is there a bugreport anywhere? | 09:38 |
| SvenKieske | and thanks :) | 09:39 |
| kevko | SvenKieske: no it is not ... normally you have router which is placed on several l3 agents right ? one is active and there is a keepalived between them where VIP is switched when active router is unresponsive ... so you have HA router right ? | 09:41 |
| SvenKieske | yes | 09:41 |
| kevko | in normal openstack deployment where you have services installed as system services ...systemd or whatever ... simply said not containerized ...you have a logs from those keepalived processes per router ... | 09:42 |
| kevko | it's because that keepalived process by default logging to /dev/log | 09:43 |
| kevko | so you can catch those messages by your syslog... | 09:43 |
| kevko | we had some problems when routers started to oscilate between hosts without the reason ... we hoped that we can see something in keepalived log ...something like ...missed healthcheck ..vip switched ...or similar | 09:44 |
| SvenKieske | Note: Kolla doesn't currently collect logs from RabbitMQ, HAProxy and Keepalived. For RabbitMQ the problem is related to RabbitMQ not having the capability to write its logs to Syslog. HAProxy and Keepalived do have that capability, but the /var/lib/kolla/dev/log Unix socket file is currently not mounted into the HAProxy and Keepalived containers. | 09:47 |
| SvenKieske | https://opendev.org/openstack/kolla-ansible/src/commit/8b0a0f72441983f43dc9a1ed777067e8c45f29e4/specs/logging-with-heka.rst | 09:47 |
| frickler | "Target Milestone for completion: Mitaka 3" nice :) | 09:48 |
| mnasiadka | Well, yes - haproxy and keepalived logging would be nice | 09:52 |
| mnasiadka | but as we can see, nobody has complained over those years | 09:53 |
| mnasiadka | and Heka is long gone :) | 09:53 |
| kevko | SvenKieske: yeah, btw I didn't see /var/lib/kolla/dev/log anywhere in a code ..just in doc | 09:53 |
| SvenKieske | usually, btw /dev/log is just a socket, nowadays it often points to /run/systemd/journal/dev-log (symlink) | 09:53 |
| kevko | mnasiadka: because everyone reading logfiles on disk :D | 09:53 |
| SvenKieske | so I suppose one solution would be to install some syslog package into a container if a package needs it? :) | 09:53 |
| mnasiadka | including me | 09:53 |
| mnasiadka | what would a syslog package help with? | 09:54 |
| mnasiadka | neutron configures keepalived to log to /dev/log | 09:54 |
| SvenKieske | doesn't keepalived require a syslog package? *looking up the dependency tree* | 09:54 |
| SvenKieske | mnasiadka: typically, afaik, if you install e.g. journald it will helpfully provide you with the standard syslog facilities, e.g. /dev/log , but I would have to double check that, afaik you can even configure that | 09:55 |
| kevko | SvenKieske: i am not sure ..but If I remember ..i was trying to mount /dev/log:/dev/log and it was not working .... | 09:55 |
| SvenKieske | my times administrating syslog servers are long gone | 09:55 |
| kevko | but still ..why keepalived is not logging anything ...just that it starts | 09:55 |
| mnasiadka | I have a feeling docker somewhat captures that | 09:55 |
| SvenKieske | kevko: yeah, because you should not mount but create it :) | 09:55 |
| mnasiadka | there's other option | 09:55 |
| SvenKieske | afaik | 09:55 |
| mnasiadka | we could start containers with docker engine stdout logging to syslog | 09:55 |
| mnasiadka | which then goes to fluentd ;-) | 09:56 |
| mnasiadka | but that won't help neutron | 09:56 |
| kevko | again ...any reason why keepalived logs only this ? https://paste.openstack.org/show/bq0Ixf8HNqshvWU7Jpap/ | 09:58 |
| SvenKieske | if you want you can directly log to fluentd from docker btw: https://docs.docker.com/config/containers/logging/fluentd/ | 09:58 |
| SvenKieske | there's a wealth of logging drivers in docker :) | 09:59 |
| SvenKieske | you should of course also be able to bindmount the socket inside the container, but beware that this does not configure all logging facilities to do what you want to do.. | 10:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 10:03 |
| SvenKieske | e.g. this user had also to mount the journald socket: https://github.com/projectatomic/atomic-site/issues/367 | 10:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 10:04 |
| SvenKieske | here is the complete blog, the "user" btw is Dan Walsh, the selinux guys :D https://projectatomic.io/blog/2016/10/playing-with-docker-logging/ | 10:06 |
| SvenKieske | kevko: you may want to also have "--flush-log-file | 10:11 |
| SvenKieske | " option in keepalived, if you really care about the logs | 10:11 |
| mnasiadka | kevko: probably the reason is in the config file or in keepalived itself ;-) | 10:11 |
| SvenKieske | well the c code in keepalived with regards to logging looks rather simple to my rather untrained eye, I have also not found any bugreports about logging not working with keepalived, but it might be the case that simply nobody noticed. | 10:12 |
| kevko | okay, it seems that it's needed to add vrrp log where it should be specified level of logging ...or type of logging state,packet,error ...etc ..chatgpt said (yeah today i am lazy :D ) ...so neutron code itself has to be tuned | 10:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: remove example certs and securityadmin_demo.sh https://review.opendev.org/c/openstack/kolla/+/919938 | 10:53 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/914191 | 11:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Fix swift disk preparation https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 11:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Force config drive for ipv6 jobs https://review.opendev.org/c/openstack/kolla-ansible/+/914185 | 11:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Rally tests and tasks https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 11:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to turn on swift tempest smoke tests https://review.opendev.org/c/openstack/kolla-ansible/+/914411 | 11:07 |
| kevko | mnasiadka: you probably forgot to change this ? https://github.com/openstack/kolla-ansible/blob/4ab17364e4e462ad7682efbe000c60bf871fcde9/zuul.d/base.yaml#L38 | 11:19 |
| kevko | mnasiadka: it should be 2024.1 no ? | 11:19 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: CI: Use 2024.1 as previous_release https://review.opendev.org/c/openstack/kolla-ansible/+/919939 | 11:21 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2024.1: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/919941 | 11:41 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Cinder and Nova: add service role https://review.opendev.org/c/openstack/kolla-ansible/+/919490 | 11:43 |
| kevko | mnasiadka: can we now merge https://review.opendev.org/c/openstack/kolla-ansible/+/914191 as caracal is released and it's again maste ? | 11:44 |
| opendevreview | Sven Kieske proposed openstack/ansible-collection-kolla stable/2024.1: Use deb822 format for Apt repositories on Ubuntu and drop apt-key https://review.opendev.org/c/openstack/ansible-collection-kolla/+/919866 | 11:49 |
| SvenKieske | just a wip, so we don't forget it ^^ | 11:51 |
| opendevreview | Adam McCartney proposed openstack/kayobe master: Add type for mtu field in docker role https://review.opendev.org/c/openstack/kayobe/+/919427 | 11:55 |
| opendevreview | Adam McCartney proposed openstack/kayobe master: Add type for mtu field in docker role https://review.opendev.org/c/openstack/kayobe/+/919427 | 12:50 |
| kevko | guys, just debugging right now some strange behaviour in nova-compute (openstack zed) ...nova-compute is stucked and when it's unstucked we can see POLLOUT ovsdb messages and some timeout ...anyone have seen this before ? containers debian ...ovs 2.15.0+ds1-8~bpo11+1 | 13:43 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: WIP: Add action for getting container names list https://review.opendev.org/c/openstack/kolla-ansible/+/905837 | 15:03 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 15:03 |
| opendevreview | Simon Dodsley proposed openstack/kolla-ansible master: Add NVMe-TCP Cinder driver support for Pure Storage https://review.opendev.org/c/openstack/kolla-ansible/+/919977 | 17:31 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 18:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!