Connect APM server to elasticsearch

I am trying to send telemetry data from ASP.NET app to elastic search. (logs, metrics, traces).

I tried to log data without APM. Logs are sent to elastic without APM, but trace does not work. To support traces I found that APM is necessary. Right now, I am trying to send all data to otel-collector which send data to APM server and APM server should send data to Elasticsearch instance.

Unfortunately, APM server could not send data to Elastic search and logs following:

{"log.level":"error","@timestamp":"2025-09-30T12:47:08.014Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

What should be fixed in this configuration to make it work ?

• If there is alternative, which excludes APM server, I would prefer that method. Including one more item in my infrastructure does not make me happy.

Infrastructure is managed by docker-compose. here is my docker compose file

services:
  elasticsearch:
    image: elasticsearch:9.1.4
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - ELASTIC_PASSWORD=Password123
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - elastic

  kibana:
    image: kibana:9.1.1
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_SERVICEACCOUNTTOKEN=AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYV9zZXJ2aWNlX3Rva2VuOmtkb3F2NnZxVEFHQXFrcEZDVDdYNlE
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=46d597016b5bdda4374240b48d07d39a
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elastic

  # 🚀 NEW SERVICE: Elastic APM Server
  apm-server:
    image: docker.elastic.co/apm/apm-server:9.0.0 # Use the same major version as Elasticsearch/Kibana
    container_name: apm-server
    environment:
      # Output to Elasticsearch
      - output.elasticsearch.hosts=["elasticsearch:9200"]
      - output.elasticsearch.username=elastic
      - output.elasticsearch.password=Password123
      # Enable OTLP intake (OpenTelemetry Collector sends data here)
      - apm-server.host=0.0.0.0:8200
      - apm-server.sampling.keep_uncategorized_services=true
    ports:
      # OTLP endpoint for the collector (OpenTelemetry)
      - "8200:8200" 
    depends_on:
      - elasticsearch
    networks:
      - elastic

  # The OpenTelemetry Collector now depends on APM Server
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    command: [--config=/etc/otelcol/config.yaml]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4317:4317" # OTLP gRPC
      - "4318:4318" # OTLP HTTP
    # ADD DEPENDENCY ON APM SERVER
    depends_on:
      - elasticsearch
      - apm-server
    networks:
      - elastic

volumes:
  es_data:
    driver: local

networks:
  elastic:
    driver: bridge

here is my otel-collector config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/elastic_apm:
    endpoint: "http://apm-server:8200"
    tls:
      insecure: true
      insecure_skip_verify: true

processors:
  batch:
    # Add these processors for better trace handling
  memory_limiter:
    limit_mib: 512
    check_interval: 1s
  resource:
    attributes:
      - key: service.instance.id
        value: ${HOSTNAME}
        action: upsert

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp/elastic_apm] 
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/elastic_apm]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/elastic_apm]