Kibana Filter to detect exceptions in Java Application logs

Question

I am trying to write a ELK-Watcher to send alert in case of any exception in my Java-Application logs.

Our application-log is already getting indexed in Elastic-Search. And under log_message field, I am able to index log-message or exception-message(in case of exception) along with exception-trace.

For example

In normal case(with no exception) log_message field will have value something like as follows.

Executing task using ExecutorService

But in case of Exception, log_message field have something like as follows

Exception while executing task : java.util.concurrent.CancellationException [StackTrace Hash 38fe72fbd18c26e8cd74b0a3c196c1441f1814e10224a323f83ec105dd355f10]
java.util.concurrent.CancellationException
        at java.util.concurrent.FutureTask.report(FutureTask.java:121)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at com.comp.tasks.TaskExecutor$MDCFuture.get(TaskExecutor.java:103)
        at com.comp.tasks.TaskExecutor.afterExecute(TaskExecutor.java:239)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Above two are just sample log_message's, but exception can originate from any other class in application with different exception-message or exception-trace.

I want to write ELK-Watcher and use Kibana-filter-query-DSL to detect if log_message field contains exceptions.

I tried many different Kibana-filter-query-dsl to filter-out log_message with exception trace. But none of them worked.

Is it possible to write any Kibana-filter-query-dsl to detect exception by looking for \n\tat in log_message field? Please help.

are you storing log.level field from your application log ? If you are storing then it will be very easy because then you need to just check if log.level value is error and send alert. — Sagar Patel
– Sagar Patel, Commented Jun 28, 2022 at 11:17
I am using log.level, but the problem is log.level=ERROR used for many other cases and not alone for Exception case. — Abhishek Sharma
– Abhishek Sharma, Commented Jun 28, 2022 at 11:51
ok got it. then you can try out to search for StackTrace or at java.lang.Thread.run which will be available for all your stacktrace. You might get some unexpected result but most of scenario it will cover. — Sagar Patel
– Sagar Patel, Commented Jun 28, 2022 at 11:53
Ok, I tried following, it works fine but it is very slow, I am doing search only for 15 min duration, but numbers of log lines generated by application in 15 min are very high( around 60K), and takes 3-4 minutes to provide result. Any solution to improve the performance of following query? { "bool": { "must": { "wildcard": { "log_message": { "value": "*at java.lang.Thread.run*" } } }, "filter": [ { "term": { "severity": "ERROR" } } ] } } — Abhishek Sharma
– Abhishek Sharma, Commented Jun 29, 2022 at 14:29
Your are trying wildcard query with prefix and postfix * thats why it is taking long time. Insted of this try match_phrase query. — Sagar Patel
– Sagar Patel, Commented Jun 30, 2022 at 5:21

Sagar Patel · Accepted Answer · 2022-06-30 05:35:33Z

0

You can try out to search for StackTrace or at java.lang.Thread.run which will be available for all your stacktrace. You might get some unexpected result but most of scenario it will cover.

Below is query you can try:

{
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "message": "at java.lang.Thread.run"
          }
        },
        {
          "match": {
            "severity": "ERROR"
          }
        }
      ]
    }
  }
}

answered Jun 30, 2022 at 5:35

Sagar Patel

5,5161 gold badge11 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Abhishek Sharma Over a year ago

Not sure why but match_phrase does not work me,

Collectives™ on Stack Overflow

Kibana Filter to detect exceptions in Java Application logs

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related