0

I am trying to visualize S3 logs in Grafana, which are collected from a load balancer. The logs are available at:

test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/year/month/date/*.gz

What is the best method to do this? Is there any tool I can use, or can it be done using Loki or InfluxDB? Please suggest, and if possible, guide me on how to set it up.

I tried using Loki, but I didn't get good results.

I have:

server1: Grafana, Loki, and Prometheus are installed.

server2: My application is running, handled by a load balancer, and the logs are collected in S3. The logs path in S3:

test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/year/month/date/*.gz

So, I tried the following steps:

Installed Promtail on server1 (where Loki is already installed). My promtail-local-config.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /data/loki/positions.yaml

clients:
  - url: http://x.x.x.x:3100/loki/api/v1/push

scrape_configs:
  - job_name: "local-s3-logs"
    static_configs:
      - targets: ['localhost']
        labels:
          job: "s3-logs"
          __path__: /var/log/s3-logs/**/*.log.gz

Sync script: My sync-s3-logs.sh:

#!/bin/bash
# Sync S3 logs to local directory

BUCKET="s3://test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/"
LOCAL_DIR="/var/log/s3-logs/"

# Create the directory if it doesn't exist
mkdir -p $LOCAL_DIR

# Sync the logs from S3 to the local directory
aws s3 sync $BUCKET $LOCAL_DIR

Running the sync script through systemd: /etc/systemd/system/s3-sync.service:

[Unit]
Description=S3 Sync Service for ELB Logs
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/opt/s3-sync/sync-s3-logs.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

/etc/systemd/system/s3-sync.timer:

[Unit]
Description=Run S3 Sync Service every 1 minute

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target

2
  • What is the best method to do this? - that's opinionated question. How one should know what is the best for you? IMHO my best: S3 trigger for any new files, which will trigger Lambda. which will parse file from S3 and ingest it into your Loki. Commented Oct 18, 2024 at 21:51
  • Agree with Jan. Look at Lambda Promtail and Loki Commented Oct 23, 2024 at 13:18

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.