8,586 questions
1
vote
0
answers
49
views
Logs not showing in Kibana
I'm unable to see the logs in Kibana. I have installed Filebeat on Kali and Elasticsearch,Logstash and Kibana on Ubuntu. I want that Filebeat should pass logs to Logstash, then Logstash should pass to ...
0
votes
0
answers
33
views
Logstash ILM Policy Causes 400 Error: Object Mapping for Field Tried to Parse as Object
I'm running an ELK stack in Docker Compose with Logstash sending logs to Elasticsearch using an ILM policy created by a separate initialization service.
Setup:
Elasticsearch with ILM policy Pong-logs-...
- 93
0
votes
1
answer
44
views
How to run logstash pipelines in Elastic Kubernetes?
We are running Elastic on Kubernetes, with Logstash running as a kind service. We're trying to integrate OutSystems logs as suggested in
https://github.com/OutSystems/outsystems-elastic-integration/...
- 1,452
0
votes
0
answers
32
views
Logstash data_stream_namespace does not work with dynamic field substitution [duplicate]
I have an ELK setup where Logstash collects logs.
In my logs I have a field like this:
{
"remote.organization.name": "navid"
}
The value of remote.organization.name can be ...
- 537
0
votes
1
answer
49
views
Kafka-Driven Elasticsearch Document Migration (Index A → B)
I'm trying to implement below steps in Logstash but getting error.
Input from kafka i.e. id
Search that Id on elastic i.e. _id
Take doc from Elastic index A
write it to another Elastic index B
Delete ...
0
votes
1
answer
39
views
Store the Kafka consumed data on disk using Logstash in case of database failure
I am consuming data from Kafka using Logstash as a consumer and want to store the data in PostgreSQL. But if in case PostgreSQL goes down then the same data should be stored on disk so that no data ...
- 422
0
votes
1
answer
65
views
Elasticsearch @timestamp field returned as current date time instead of from log message
This is my logstash.conf file:
filter {
grok {
match => {
"message" => '%{IPV4:client_ip} - - \[%{HTTPDATE:log_timestamp}\] "%{WORD:method} %{URIPATHPARAM:uri} HTTP/%{...
- 1,452
1
vote
1
answer
69
views
build a logstash image with a plugin generates an error
I'm trying to create my own image of logstash, in order to add a plugin:
FROM docker.elastic.co/logstash/logstash:8.1.0
RUN bin/logstash-plugin install logstash-filter-translate
The generation works ...
- 2,618
0
votes
1
answer
31
views
Export all logs from ELK stack to external destination for compliance purpose
I am writing to you because I would need to export logs from inside elk to outside, like to blob in azure or any other destination pointc, for compliance purpose. Do you know any solution available?
...
- 1
0
votes
1
answer
85
views
Logstash Sending Logs to Splunk with Incorrect Timestamp
I am currently posting logs to Splunk using the Logstash HTTP output plugin. However, I have encountered an issue where my logs contain timestamps in CST (Central Standard Time), but when they are ...
0
votes
0
answers
41
views
Logstash/ Poco::Logger - JSON parse error
I am trying to debug this issue showing up related to Json Parsing. My application is using POCO::Logger for logging which is rotating the log files hourly basis. During the rotation, the Logstash is ...
0
votes
0
answers
45
views
How to close a sql server db connection for logstash pipeline when elasticsearch is unreachble
I am using Logstash 8.12 My logstash pipeline reads data from sql server and send to Azure ElasticSearch.
It works fine when ES is available. When Elastic search node is unreachable, I get the error ...
- 85
2
votes
1
answer
51
views
how to remove /r from field value in logstash grok
We have this kind of logs collected from Winlogbeat.
AlertName=Wireless Access Point Alert
AlertStatus=Active
AlertActiveID=8618424
AlertDescription=
Severity=Critical
Here “AlertName” keeps ...
- 39
1
vote
1
answer
35
views
Logstash field is never shown after aggregation
I have logstash version 7.8.0
Can someone tell me why the aggregation below never shown THREAD_ID field into documents please ?
My field : thread_id is added in the end of aggregation ..
Sample.log:
...
- 49
1
vote
0
answers
316
views
How to consume Kafka messages from Logstash?
I'm playing a bit with the latest versions of Logstash and Kafka but I can't get the Kafka input to work.
Here a brief summary of my setup:
I'm using Docker Compose with apache/kafka:3.9.0 and ...
- 3,469
0
votes
1
answer
88
views
while using aggregate plugin in logstash script i am getting undefined local variable or method 'map'
i am using aggregate plugin to keep count of how many events happened for a particular task id but when i try to use the map values inside the timeout_code i am getting "undefined local variable ...
- 1
1
vote
1
answer
210
views
Filebeat - Log Processing Issues/Delay/Data Loss
We are experiencing significant challenges with log processing on three of our hosts. Each of these hosts runs nine services, generating between 30,000 to 72,000 events per minute per log file. The ...
1
vote
0
answers
225
views
Spring Boot application Docker Compose Logstash connection issue
I'm trying to run a Spring Boot application using Docker Compose, which includes MongoDB, Elasticsearch, Kibana, and Logstash. However, when my Spring Boot app starts, it can't connect to Logstash and ...
- 1,176
0
votes
0
answers
174
views
Cannot authenticate to MSSQL database using AD credentials via jdbc driver
We use mssql-jdbc-12.6.4 in Logstash to connect to MSSQL database, but we are facing authentication problem: Login failed for user 'domian\username'.
Here is the traceback:
Configuration with jdbc ...
- 81
0
votes
0
answers
30
views
Logstash and parsing submessage
I have 3 types of log messages
2024-07-22 11:45:29.125 +02:00 [Information] [EventService] Events generation finished
2024-07-22 11:45:29.125 +02:00 [Information] [Process] Result: "[11:45:29 INF]...
- 4,248
0
votes
0
answers
58
views
Logrotation and new entries with timestamps from already rotated file
I have a file, metrics.log, that gets log rotated daily at 23:59 via log4j's RollingFileAppender. The log rotated file is kept in another directory as metrics-$DATE.log.
It contains JSON-formatted log ...
- 21
0
votes
1
answer
32
views
adding date in elasticsearch index name doesn't work
I have been using the following code for the elasticsearch output plugin of a logstash, but it creates the index named as mt-raw-00001
output {
elasticsearch {
hosts =>...
0
votes
0
answers
40
views
Aggregate logstash message from multiple entries into only one (stack trace)
I've been struggling to find a solution to this problem. Maybe someone can shed a light on it.
All of our containers (Openshift) messages are posted to a Kafka topic, which feeds Logstash with json ...
- 3,735
0
votes
0
answers
38
views
Use a .properties file to pass configuration variables to Logstash
I am setting Logstash on Docker, and the deployment chart gets some environment variables and places them in a file secrets.properties inside the container,
What I want to do is use this file to pass ...
- 90
0
votes
1
answer
65
views
Problems with updating :sql_last_value
I want to synchronize data between postgres and elasticsearch for this I am using logstash.
This is how looks configure file for logstash
input {
jdbc {
jdbc_connection_string => "...
- 3