| *** zzzeek has quit IRC | 00:00 | |
| *** zzzeek has joined #openstack-infra | 00:00 | |
| *** yamamoto has quit IRC | 00:11 | |
| *** yamamoto has joined #openstack-infra | 00:20 | |
| *** paladox has quit IRC | 00:21 | |
| *** zzzeek has quit IRC | 00:22 | |
| *** tosky has quit IRC | 00:23 | |
| melwitt | clarkb: do you happen to know where this prefix in all the logs comes from? "Jan 22 20:53:33.660855 ubuntu-focal-vexxhost-ca-ymq-1-0022678509 nova-compute[86740]:" | 00:24 |
|---|---|---|
| *** zzzeek has joined #openstack-infra | 00:24 | |
| melwitt | I'm so far failing to find where. it doesn't align with any of the oslo.log prefix conf options I find in the gate | 00:24 |
| fungi | you mean the "ubuntu-focal-vexxhost-ca-ymq-1-0022678509" part? that's the hostname, syslog generally includes it | 00:25 |
| melwitt | this is in the nova-compute logs | 00:25 |
| melwitt | it's on everything afaict | 00:25 |
| fungi | format is roughly: timestamp hostname process[pid]: message | 00:26 |
| melwitt | that entire line is prefixed on everything and I can't tell where the heck it's coming from. what is doing it | 00:26 |
| melwitt | yeah | 00:26 |
| melwitt | I'm on a quest to restore multiline indexing for exception tracebacks in the gate logs and so far I've found it's because this line prefix is different than it was in the past (> 1 year ago) | 00:27 |
| fungi | not sure what would decorate the nova logs with it, but that's syslog format... maybe we have something set to make openstack service logs look like syslog for consistency of parsing? | 00:27 |
| fungi | is devstack logging into journald now and we're dumping syslog-like filtered logs from there? | 00:28 |
| melwitt | and I want to understand where it's coming from so I am more informed about what needs to change in our logstash multiline filters and document nicely in the commit message | 00:28 |
| fungi | i want to say we stopped running services under gnu screen and creating hardcopy dumps from it years back, i really don't recall how long ago | 00:28 |
| melwitt | I dunno and I'm not doing a great job at finding out yet. so I thought I'd ask before I continue my clumsy searching, in case you or clark just already knew :) | 00:29 |
| melwitt | yeah we started running under systemd awhile back instead of screen | 00:29 |
| fungi | my next search would probably be to grep the devstack repo for "journal" | 00:29 |
| melwitt | thanks, any hint helps haha | 00:29 |
| fungi | my guess is it's formatting applied to the records extracted from the systemd journal | 00:30 |
| melwitt | ahhh... that would make sense | 00:30 |
| fungi | likely by whatever tool (journalctl?) is being used for that | 00:30 |
| melwitt | yeah journalctl | 00:30 |
| fungi | output format is probably configurable in its cli | 00:30 |
| melwitt | I literally never noticed before if it had a different format. guh | 00:30 |
| *** jamesmcarthur has joined #openstack-infra | 00:31 | |
| fungi | i still miss bad old (r)syslog(-ng) and straight up text files. it's taking me times to come to term with my grief | 00:32 |
| *** paladox has joined #openstack-infra | 00:32 | |
| fungi | er, time to come to terms, that is | 00:32 |
| melwitt | heh | 00:32 |
| * fungi goes back to shaking his cane at the kids on his lawn | 00:34 | |
| melwitt | thanks for the help :) | 00:34 |
| fungi | yw! here to provide vague clues and no answers to any other questions you might have too | 00:34 |
| clarkb | melwitt: fungi its journald/syslog | 00:35 |
| clarkb | I think journald now ya | 00:35 |
| melwitt | lol, vague clues save me a lot of time tbh | 00:35 |
| fungi | awesome, so my wild stabs in the dark weren't too far off the mark | 00:36 |
| melwitt | :D | 00:36 |
| clarkb | when we put the services into systemd units we got journald logging for free which made for easy log collection | 00:36 |
| melwitt | ahhh so I think all we need to do is update the logstash filters to sync up with that | 00:37 |
| *** jamesmcarthur has quit IRC | 00:37 | |
| *** jamesmcarthur has joined #openstack-infra | 00:37 | |
| clarkb | it should've been updated for that iirc | 00:37 |
| fungi | likely there's a filter type of "syslog" or similar | 00:37 |
| clarkb | oh for multiline? ya maybe that didnt | 00:37 |
| melwitt | I've been having a really hard time trying to get my logstash queries dialed in because I can't take different pieces of a multiline traceback | 00:37 |
| melwitt | yeah, I was looking at this https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf#L10-L40 | 00:38 |
| melwitt | and I can see why it doesn't work | 00:39 |
| melwitt | it's using markers for a format we no longer have | 00:39 |
| fungi | and the basic syslog filter we've defined likely isn't the answer either because it doesn't look multi-line aware | 00:40 |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** paladox has quit IRC | 01:33 | |
| *** jamesmcarthur has quit IRC | 01:37 | |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** paladox has joined #openstack-infra | 01:38 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** jamesmcarthur has joined #openstack-infra | 01:48 | |
| *** armax has quit IRC | 02:17 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** jamesmcarthur has joined #openstack-infra | 02:29 | |
| *** rcernin has quit IRC | 02:33 | |
| *** zzzeek has quit IRC | 02:35 | |
| *** jamesmcarthur has quit IRC | 02:36 | |
| *** zzzeek has joined #openstack-infra | 02:37 | |
| *** jamesmcarthur has joined #openstack-infra | 02:52 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** matt_kosut has joined #openstack-infra | 03:00 | |
| *** matt_kosut has quit IRC | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** jamesmcarthur has joined #openstack-infra | 03:07 | |
| *** dave-mccowan has quit IRC | 03:10 | |
| *** rcernin has joined #openstack-infra | 03:14 | |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** yamamoto has quit IRC | 03:19 | |
| *** sboyron has quit IRC | 03:22 | |
| *** jamesdenton has quit IRC | 03:24 | |
| *** jamesden_ has joined #openstack-infra | 03:25 | |
| openstackgerrit | melanie witt proposed openstack/logstash-filters master: Add multiline filtering for journald based tracebacks https://review.opendev.org/c/openstack/logstash-filters/+/772145 | 03:36 |
| *** yamamoto has joined #openstack-infra | 03:36 | |
| *** yamamoto has quit IRC | 03:39 | |
| *** jamesmcarthur has joined #openstack-infra | 03:53 | |
| *** jamesmcarthur has quit IRC | 03:53 | |
| *** rcernin has quit IRC | 04:05 | |
| *** zzzeek has quit IRC | 04:19 | |
| *** zzzeek has joined #openstack-infra | 04:20 | |
| *** jamesden_ has quit IRC | 04:29 | |
| *** jamesdenton has joined #openstack-infra | 04:30 | |
| *** lbragstad_ has joined #openstack-infra | 04:32 | |
| *** lbragstad has quit IRC | 04:34 | |
| fungi | clarkb: ^ that has to deploy to the logstash workers to take effect at ingest, right? | 04:59 |
| *** ociuhandu has joined #openstack-infra | 05:24 | |
| *** ociuhandu has quit IRC | 05:28 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| *** gyee has quit IRC | 05:46 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** sshnaidm|afk has quit IRC | 05:55 | |
| *** sshnaidm|afk has joined #openstack-infra | 05:57 | |
| *** bdodd has quit IRC | 06:48 | |
| *** bdodd has joined #openstack-infra | 06:49 | |
| *** pots has quit IRC | 06:50 | |
| *** bdodd_ has joined #openstack-infra | 06:56 | |
| *** bdodd has quit IRC | 06:56 | |
| *** matt_kosut has joined #openstack-infra | 07:01 | |
| *** matt_kosut has quit IRC | 07:05 | |
| *** lbragstad_ has quit IRC | 07:18 | |
| *** d34dh0r53 has quit IRC | 07:29 | |
| *** d34dh0r53 has joined #openstack-infra | 07:31 | |
| *** dklyle has quit IRC | 07:45 | |
| *** jamesdenton has quit IRC | 07:50 | |
| *** jamesdenton has joined #openstack-infra | 07:50 | |
| *** jamesdenton has quit IRC | 08:24 | |
| *** jamesden_ has joined #openstack-infra | 08:24 | |
| *** rcernin has joined #openstack-infra | 08:26 | |
| *** rcernin has quit IRC | 08:31 | |
| *** rcernin has joined #openstack-infra | 08:50 | |
| *** xek_ has joined #openstack-infra | 08:56 | |
| *** matt_kosut has joined #openstack-infra | 09:01 | |
| *** zxiiro has quit IRC | 09:04 | |
| *** PrinzElvis has quit IRC | 09:04 | |
| *** masayukig has quit IRC | 09:04 | |
| *** zigo has quit IRC | 09:04 | |
| *** sorrison has quit IRC | 09:04 | |
| *** matt_kosut has quit IRC | 09:06 | |
| *** zxiiro has joined #openstack-infra | 09:06 | |
| *** PrinzElvis has joined #openstack-infra | 09:06 | |
| *** masayukig has joined #openstack-infra | 09:06 | |
| *** zigo has joined #openstack-infra | 09:06 | |
| *** sorrison has joined #openstack-infra | 09:06 | |
| *** Jeffrey4l has quit IRC | 09:08 | |
| *** bdodd_ has quit IRC | 09:08 | |
| *** Jeffrey4l has joined #openstack-infra | 09:09 | |
| *** mordred has quit IRC | 09:10 | |
| *** bdodd has joined #openstack-infra | 09:10 | |
| *** wolsen has quit IRC | 09:12 | |
| *** JanZerebecki[m] has quit IRC | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:37 | |
| *** lifeless has quit IRC | 09:45 | |
| *** lifeless has joined #openstack-infra | 09:46 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** JanZerebecki[m] has joined #openstack-infra | 10:21 | |
| *** ociuhandu has joined #openstack-infra | 10:22 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| *** ociuhandu has quit IRC | 10:26 | |
| *** jamesden_ has quit IRC | 10:37 | |
| *** jamesdenton has joined #openstack-infra | 10:37 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** matt_kosut has joined #openstack-infra | 10:42 | |
| *** matt_kosut has quit IRC | 10:46 | |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** rcernin has quit IRC | 11:00 | |
| *** mordred has joined #openstack-infra | 11:01 | |
| *** wolsen has joined #openstack-infra | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:16 | |
| *** rcernin has joined #openstack-infra | 11:22 | |
| *** rcernin has quit IRC | 11:37 | |
| *** yamamoto has quit IRC | 11:39 | |
| *** rcernin has joined #openstack-infra | 12:03 | |
| *** rcernin has quit IRC | 12:09 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** rcernin has joined #openstack-infra | 12:44 | |
| *** rcernin has quit IRC | 12:49 | |
| *** xek_ has quit IRC | 12:54 | |
| *** tosky has joined #openstack-infra | 13:01 | |
| *** rcernin has joined #openstack-infra | 14:07 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** rcernin has quit IRC | 14:25 | |
| *** yamamoto has quit IRC | 14:31 | |
| *** jamesdenton has quit IRC | 14:46 | |
| *** jamesdenton has joined #openstack-infra | 14:47 | |
| *** tosky has quit IRC | 15:09 | |
| *** tosky has joined #openstack-infra | 15:18 | |
| *** HenryG has quit IRC | 16:08 | |
| *** HenryG has joined #openstack-infra | 16:09 | |
| *** tosky has quit IRC | 16:42 | |
| *** jamesdenton has quit IRC | 17:24 | |
| *** tosky has joined #openstack-infra | 18:03 | |
| *** tosky_ has joined #openstack-infra | 18:08 | |
| *** tosky has quit IRC | 18:08 | |
| *** tosky_ is now known as tosky | 18:16 | |
| *** lbragstad_ has joined #openstack-infra | 18:54 | |
| *** lbragstad_ has quit IRC | 18:59 | |
| *** zxiiro has quit IRC | 19:07 | |
| *** bdodd has quit IRC | 19:29 | |
| *** bdodd has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has joined #openstack-infra | 19:52 | |
| *** sshnaidm|afk has quit IRC | 20:14 | |
| *** zzzeek has quit IRC | 20:16 | |
| *** zzzeek has joined #openstack-infra | 20:19 | |
| *** sshnaidm|afk has joined #openstack-infra | 20:27 | |
| *** yamamoto has joined #openstack-infra | 20:29 | |
| *** yamamoto has quit IRC | 20:34 | |
| *** jamesmcarthur has quit IRC | 20:35 | |
| *** jamesmcarthur has joined #openstack-infra | 20:36 | |
| *** jamesmcarthur has quit IRC | 20:41 | |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| *** sshnaidm|afk has quit IRC | 20:46 | |
| *** jamesmcarthur has quit IRC | 20:46 | |
| *** sshnaidm has joined #openstack-infra | 20:51 | |
| *** jamesmcarthur has joined #openstack-infra | 21:01 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| *** rcernin has joined #openstack-infra | 21:07 | |
| *** bdodd has quit IRC | 21:08 | |
| *** bdodd has joined #openstack-infra | 21:11 | |
| *** rcernin has quit IRC | 21:14 | |
| *** jamesmcarthur has joined #openstack-infra | 21:23 | |
| *** dave-mccowan has joined #openstack-infra | 21:41 | |
| *** dave-mccowan has quit IRC | 21:44 | |
| *** jamesmcarthur has quit IRC | 22:02 | |
| *** jamesmcarthur has joined #openstack-infra | 22:05 | |
| *** jamesmcarthur has quit IRC | 22:10 | |
| *** yamamoto has joined #openstack-infra | 22:25 | |
| *** jamesmcarthur has joined #openstack-infra | 22:33 | |
| *** jamesmcarthur has quit IRC | 22:36 | |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| *** jamesmcarthur has quit IRC | 23:00 | |
| *** jamesmcarthur has joined #openstack-infra | 23:00 | |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** jamesmcarthur has joined #openstack-infra | 23:06 | |
| *** yamamoto has quit IRC | 23:18 | |
| *** yamamoto has joined #openstack-infra | 23:21 | |
| *** dklyle has joined #openstack-infra | 23:27 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!