Local logstash pipeline (kafka input plugin) hangs at "Setting offset for partition <topic-name>-0 to the committed offset FetchPosition"

Question

Every time I launch my logstash pipeline, the pipeline hangs and does not do anything nor does it time out or produce errors. This is as far as it gets before it just stops.

[INFO ][org.apache.kafka.clients.consumer.internals.SubscriptionState][main][8527eb95a42a27f4f71b2bd0fa1953d2bb496388148b0cfcc9c55617f74fcf65] [Consumer clientId=logstash-0, groupId=logstash] Resetting offset for partition demotest2-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.1.187:9092 (id: 0 rack: null)], epoch=0}}.

On my local machine, I am running the following:

zookeeper
kafka server (kafka_2.12-3.2.1) on default 0.0.0.0:9092
a producer, created some messages in console to upstream topic
attempting a logstash pipeline (version 8.3.3)
Below is my pipeline conf

input {
 kafka
 {
  bootstrap_servers => "0.0.0.0:9092"
  topics => "demotest2"
 }
}
output {
  file {
   path => "/Users/chungking007/Documents/2022/logstash/logs/out.log"
  }
}

Is there any other way to view logs of pipelines that get stuck like this? I have verified that the kafka topic can both produce & consume topics through the console, and independently I have verified that logstash can read input and output via console, or write to a file as well.

007chungking · Accepted Answer · 2022-08-19 16:02:34Z

1

I resolved this myself by passing the following into the input. Considering this is only for local development, my method is merely to create a new consumer and group for each iteration as I test my filter. This offset reset seems to resolve the issue with offsets once pipeline is launched, and it suits my purposes for now

  topics => "accurate-topic"
  group_id => "new_consumer_aug19_7"
  client_id => "new_consumer_aug19_7"
  consumer_threads => 1
  auto_offset_reset => "earliest"
 }

answered Aug 19, 2022 at 16:02

007chungking

556 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Local logstash pipeline (kafka input plugin) hangs at "Setting offset for partition <topic-name>-0 to the committed offset FetchPosition"

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related