elasticsearch message field value

Question

I am sending json messages to logstash getting indexed by elasticsearch and managed to setup the UI dashboard in Kibana. I would like to filter the data by the message fields and cannot figure out how or where to do this. An example of my message:

{"message":"{"pubDate":"2014-02-25T13:09:14",
 "scrapeDate":"2014-02-5T13:09:26",
 "Id":"78967",
 "query":"samsung S5",
 "lang":"en"}

Right now it counts all these messages coming in but I need to get each message filtered by the fields itself for example like Id or lang or query. Does this have to be done in the config file or can it be created in Kibana interface.

Ban-Chuan Lim · Accepted Answer · 2014-02-26 01:39:13Z

4

First, I assume your json messages is

{
   "pubDate":"2014-02-25T13:09:14",
   "scrapeDate":"2014-02-5T13:09:26",
   "Id":"78967",
   "query":"samsung S5",
   "lang":"en"
}

When you send your message to logstash, you need to specify the codec to json. As show in the configuration below:

input {
    stdin {
            codec => json
    }
}

output {
    elasticsearch {
            cluster => "abc"
    }
}

Logstash will parsing your message to different field, like the output:

{
   "pubDate" => "2014-02-25T13:09:14",
"scrapeDate" => "2014-02-5T13:09:26",
        "Id" => "78967",
     "query" => "samsung S5",
      "lang" => "en",
  "@version" => "1",
"@timestamp" => "2014-02-26T01:36:15.336Z",
      "host" => "AAAAAAAAAA"
 }

When you show this data in Kibana, You can use fieldname:value to query and filter what you need. For example, you can query all message with lang:en.

answered Feb 26, 2014 at 1:39

Ban-Chuan Lim

7,8904 gold badges37 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

vbNewbie Over a year ago

Thanks for your response. Appreciate it. I changed the config file as you suggested but not seeing any data come through. I will figure it out but one more question if you don't mind. Where are the log messages stored so that I can clean up that directory when it gets too full.

vbNewbie Over a year ago

also would the conf file not have tcp instead of stdin right?

vbNewbie Over a year ago

would still appreciate any response about the log cleanup

Ban-Chuan Lim Over a year ago

After Logstash parses the log, the log data will send to elasticsearch and store in it! you can visit $elasticsearch/data. You can delete all the data there or use elasticsearch API to delete it. elasticsearch.org/guide/en/elasticsearch/reference/current/…

Collectives™ on Stack Overflow

elasticsearch message field value

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related