Logstash output works but fail to feed elasticsearch

Question

I am trying to pass my API custom log files to Kibana. For that I use myLogFiles.log -> Logstash -> Elasticsearch -> Kibana.

PROBLEM

Logstash output tells that everything is fine, but Elasticsearch remain empty (I checked that with kibana or elasticsearch web visualisation plugins)

QUESTION

How can I feed elasticsearch to visualise my data in kibana ?

DEBUG SO FAR

To have some debug I told Logstash to ouput in a .log file and on console. see my logstash.conf:

input {
    file {
        path => '/home/***/dev_logstach/acci.log'
    start_position => beginning
    }
}


filter {
    grok {
        match => {"message" =>  "%{WORD:key} %{WORD:userID} %{WORD:lakeID} %{WORD:ballID} %{NUMBER:longitude} %{NUMBER:latitude} %{TIMESTAMP_ISO8601:date_evenement}"}
    }
}


output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "testACCI"
  }
  stdout { codec => rubydebug }
  file {
   path => "/home/***/dev_logstach/testLog.log"
   create_if_deleted => true
  }
}

And a logstash output

{
            "userID" => "HJcOX",
        "@timestamp" => 2018-04-10T13:27:47.723Z,
          "latitude" => "-123.8",
           "message" => "{message:ACWWCI HJcOX qHYFM ABCCCC -22.5 -123.8 2018-04-10T09:11:06.173Z,level:info}",
              "path" => "/home/***/dev_logstach/acci.log",
    "date_evenement" => "2018-04-10T09:11:06.173Z",
          "@version" => "1",
            "lakeID" => "qHYFM",
               "key" => "ACWWCI",
         "ballID" => "ABCCCC",
              "host" => "sd-10****",
         "longitude" => "-22.5"
}

When I run again it to not log previous data, and when logstash run if I

 echo -e "new data line" >> acci.log

the new data appear in the logs. I then suppose that the data are send somewhere but I have no clue where and how can I move forward.

In elasticsearch log I just see a warning:

2018-04-10T17:01:52,507][WARN ][o.e.d.i.m.MapperService  ] [_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type

P.S. I read some problem like that but it was due to date format mine is valide (YYYY-MM-DD...)

Evaldas Buinauskas · Accepted Answer · 2018-04-11 09:53:00Z

0

Couple of options:

Create your elasticsearch index in advance
Create elasticsearch index template

This should resolve your issue.

answered Apr 11, 2018 at 9:53

Evaldas Buinauskas

14.2k11 gold badges63 silver badges119 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

lu1her Over a year ago

Creating the index seems to fix my problem but I don t have the data yet in kibana, I think it will works with an elasticsearch index template, as soon as I test it I ll give a feed back.

lu1her Over a year ago

Just creating the index in elasticsearch was enought

Evaldas Buinauskas Over a year ago

Yeah, both index or index templates should work. Logstash was defaulting to its' own template most likely, which is still outdated.

Collectives™ on Stack Overflow

Logstash output works but fail to feed elasticsearch

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related