I have a tomcat installation which has a few Java Spring Boot application deployed. This causes two different log pattern in the catalina.out logfile: Default tomcat logging:
18-Nov-2024 14:15:16.898 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/connector#api.war] has finished in [3,731] ms
and Spring Boot logging:
2024-12-03T09:04:40.762Z WARN 865 --- [o-8443-exec-247] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'GET' is not supported]
Because Boot Spring sometimes has an empty line between the beginning a log message and the corresponding stacktrace the multiline parser "java" does not work. Thats why I have integrated the following configuration:
fluent-bit.conf:
[SERVICE]
flush 1
daemon Off
log_level debug
parsers_file parsers.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
[FILTER]
Name record_modifier
Match *
Record hostname ${HOSTNAME}
[INPUT]
name tail
path /tmp/catalina.out
Tag tomcatLog
multiline.parser multiline-tomcat-default, multiline-tomcat-springboot
[FILTER]
Name Parser
Match tomcatLog
Key_name log
parser tomcat-springboot
parser tomcat-default
[OUTPUT]
name stdout
match *
parsers.conf:
[PARSER]
Name tomcat-default
Format regex
Regex ^(?<time>(\d{2}-\w*-\d{4}\s[\d|:|.]*))\s(?<level>\w*)\s(?<message>.*(?:\r?\n.*)*)
Time_Key time
Time_Format %d-%b-%Y %H:%M:%S.%L
[MULTILINE_PARSER]
Name multiline-tomcat-default
Type regex
flush_timeout 1000
rule "start_state" "/^(?<time>(\d{2}-\w*-\d{4}\s[\d|:|.]*))\s(?<level>\w*)\s(?<message>.*)$/" "cont"
rule "cont" "/(^\s*$)|(^\s.*$)/" "cont"
[PARSER]
Name tomcat-springboot
Format regex
Regex ^(?<time>(\d|-|:|\.|T|Z)*)\s*(?<level>\w*?)\s(?<pid>\d*)\s-{3}\s\[(?<threadName>.*)\]\s(?<class>.*)\s:\s(?<message>.*(?:\r?\n.*)*)
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
[MULTILINE_PARSER]
Name multiline-tomcat-springboot
Type regex
flush_timeout 1000
rule "start_state" "/^(?<time>(\d|-|:|\.|T|Z)*)\s*(?<level>\w*?)\s(?<pid>\d*)\s-{3}\s\[(?<threadName>.*)\]\s(?<class>.*)\s:\s(?<message>.*)/" "cont"
rule "cont" "/^\D.*$/" "cont"
The default messages get parsed successfully and the temporary key "log" gets removed. When parsing a spring message I can see one successfully parsed message and one with the unparsed message:
[0] tomcatLog: [[1733216505.546000000, {}], {"level"=>"INFO", "pid"=>"865", "threadName"=>"o-8443-exec-234", "class"=>"o.s.web.servlet.DispatcherServlet ", "message"=>"Completed initialization in 2 ms
"}]
[1] tomcatLog: [[1733216680.757000000, {}], {"level"=>"WARN", "pid"=>"865", "threadName"=>"o-8443-exec-247", "class"=>".m.m.a.ExceptionHandlerExceptionResolver", "message"=>"Failure in @ExceptionHandler com.falke.rdmconnector.rest.GlobalExceptionHandler#handleRuntimeException(RuntimeException)
java.lang.IllegalStateException: Could not resolve parameter [0] in org.springframework.web.ErrorResponse com.falke.rdmconnector.rest.GlobalExceptionHandler.handleRuntimeException(java.lang.RuntimeException): No suitable resolver
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:221) ~[spring-web-6.1.12.jar:6.1.12]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:178) ~[spring-web-6.1.12.jar:6.1.12]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.1.12.jar:6.1.12]
at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:432) ~[spring-webmvc-6.1.12.jar:6.1.12] at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:74) ~[spring-webmvc-6.1.12.jar:6.1.12]
"}]
[2] tomcatLog: [[1733395924.011246623, {}], {"log"=>" at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:221) ~[spring-web-6.1.12.jar:6.1.12]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:178) ~[spring-web-6.1.12.jar:6.1.12]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.1.12.jar:6.1.12]
at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:432) ~[spring-webmvc-6.1.12.jar:6.1.12] at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:74) ~[spring-webmvc-6.1.12.jar:6.1.12]
", "hostname"=>"appdev"}]
[3] tomcatLog: [[1733216680.762000000, {}], {"level"=>"WARN", "pid"=>"865", "threadName"=>"o-8443-exec-247", "class"=>".w.s.m.s.DefaultHandlerExceptionResolver", "message"=>"Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'GET' is not supported]"}]
My suspect is that the first (parsed) message is from the first "parser" entry the seconds one is untouched because the second entry fails.
So my question is: How do I parse two different multiline log formats from one source?