I am trying to visualize S3 logs in Grafana, which are collected from a load balancer. The logs are available at:
test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/year/month/date/*.gz
What is the best method to do this? Is there any tool I can use, or can it be done using Loki or InfluxDB? Please suggest, and if possible, guide me on how to set it up.
I tried using Loki, but I didn't get good results.
I have:
server1: Grafana, Loki, and Prometheus are installed.
server2: My application is running, handled by a load balancer, and the logs are collected in S3. The logs path in S3:
test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/year/month/date/*.gz
So, I tried the following steps:
Installed Promtail on server1 (where Loki is already installed). My promtail-local-config.yaml:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /data/loki/positions.yaml
clients:
- url: http://x.x.x.x:3100/loki/api/v1/push
scrape_configs:
- job_name: "local-s3-logs"
static_configs:
- targets: ['localhost']
labels:
job: "s3-logs"
__path__: /var/log/s3-logs/**/*.log.gz
Sync script: My sync-s3-logs.sh:
#!/bin/bash
# Sync S3 logs to local directory
BUCKET="s3://test-bkp/AWSLogs/851725603941/elasticloadbalancing/us-east-2/"
LOCAL_DIR="/var/log/s3-logs/"
# Create the directory if it doesn't exist
mkdir -p $LOCAL_DIR
# Sync the logs from S3 to the local directory
aws s3 sync $BUCKET $LOCAL_DIR
Running the sync script through systemd: /etc/systemd/system/s3-sync.service:
[Unit]
Description=S3 Sync Service for ELB Logs
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
ExecStart=/opt/s3-sync/sync-s3-logs.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
/etc/systemd/system/s3-sync.timer:
[Unit]
Description=Run S3 Sync Service every 1 minute
[Timer]
OnBootSec=1min
OnUnitActiveSec=1min
[Install]
WantedBy=timers.target
What is the best method to do this?- that's opinionated question. How one should know what is the best for you? IMHO my best: S3 trigger for any new files, which will trigger Lambda. which will parse file from S3 and ingest it into your Loki.