how can i store in two index using two JSON formated log files using filebeat and output to elasticsearch

Question

below is my configuration file for filebeat which is present in /etc/filebeat/filebeat.yml, it throws an error of

Failed to publish events: temporary bulk send failure

filebeat.prospectors:
  - paths:
     - /var/log/nginx/virus123.log
   input_type: log
   fields:
      type:virus123
   json.keys_under_root: true


   - paths:
     - /var/log/nginx/virus1234.log
     input_type: log
     fields:
       type:virus1234
     json.keys_under_root: true

setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
setup.template.overwrite: true

processors:
 - drop_fields:
     fields: ["beat","source"]


output.elasticsearch:
  index: index: "filebeat-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

  hosts: ["http://127.0.0.1:9200"]

Also, normally the bulk sense failure is caused by an error on the Elasticsearch side. Knowing which error elasticsearch returns could be helpful here. — Green
– Green, Commented Jun 16, 2018 at 14:30

Green · Accepted Answer · 2018-06-16 14:34:17Z

1

I think I found your problem, Although i'm not sure it is the only problem

index: index: "filebeat-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

should be:

index: "filebeat-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

I saw a similar problem with a wrong index which cause the same error that you showed

answered Jun 16, 2018 at 14:34

Green

2,5653 gold badges27 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Virendra Singh Over a year ago

that was a typo mistake it's not working solution, I go with logagent it's super awesome and lightweight as compared to logstash

Collectives™ on Stack Overflow

how can i store in two index using two JSON formated log files using filebeat and output to elasticsearch

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related