Ingesting logs from Azure Blob Storage to Loki

Question

Similar to a previous question related to reading logs from Azure Storage Accounts for Fluentd/Fluentbit, I am searching for a more direct way to read logs from Azure Storage Accounts into Loki.

When searching, I found Amazon S3 plugins for Fleuntd/bit and Logstash, nothing for Azure Storage Accounts. Only Sumo Logic seems to support streaming logs from azure storage accounts.

Here seems to be a possibility to read Azure logs into Loki from Azure Event Hubs

I have implemented Java Azure Functions that get triggered when a storage event fires on the containers of the Azure Storage Account. Then, I am going to process and push the logs from the log files (in Json) to Loki via its API, or perhaps use a Java client help me with the pushing of the log lines to Loki.

My question is if someone got a better idea, like having plug-ins similar to the ones provided for Amazon S3 from Fluentbit/Fluentd or Logstash. When possible, I want to avoid using Azure Functions for Java.

Also, would it be better (when using Loki) to get the logs sent to an Azure Event Hub rather than Storage Accounts? The cost factor is critical for me, therefore I first opted to using Storage Accounts and not event hubs.

for now, only this way to read logs from Azure Storage Accounts into Loki. — P Sampath
– P Sampath, Commented Mar 1, 2024 at 3:29
Thanks a lot. I am comparing between the use of azure functions with blob triggers and the use of azure event hubs. Both seem to require a lot of manual processing (e.g., parsing JSON files, validating the time of the logs, and making sure they are sent to Loki in the right order). I still experiment with the event-hub option to see if it will give better results. I was hoping to see if someone successfully used the Logstash plugin or some other ways. All available plugins seem to be tailord for amazon S3. — kloudkid
– kloudkid, Commented Mar 1, 2024 at 8:54

kloudkid · Accepted Answer · 2024-05-06 08:49:46Z

Following Naveen's advice, I firstly used Event Subscriptions and system topics (Azure Event Grid) to receive events when blob files get added to storage accounts.

Configuring the "azure_event_hubs" as above allowed Promtail to forward logs to Loki whenever blob events trigger, but this meant I did not get the content of the logs stored as JSON files on the storage containers.

In order to get the Azure diagnostic log entries, I chose "Stream to an event hub" option in Azure Diagnostic settings: eh as destination for diagnostic logs Here you can specify which log categories you are interested in.

Then I adjusted the Promtail configuration as follows:

- job_name: azure_event_hubs
azure_event_hubs:
  fully_qualified_namespace: ehns.servicebus.windows.net:9093
  connection_string: connection-string
  event_hubs: eh-name
  labels:
    job: azure_event_hub
relabel_configs:
  - action: replace
    source_labels:
      - __azure_event_hubs_category
    target_label: category

as in promtail config

I did not need a forward_to attribute, because I do not use Grafana flow.

I am now able to receive Azure Diagnostic Logs directly from Azure in Loki and can query them in Grafana with "categroy" and "job" as labels.

Under Promtail, azure_event_hubs can now be seen as a target: promtail targets

The only disadvantage to this is that basic pricing plan Azure Event Hubs cannot be used, but rather standard pricing and above.

While this approach solves my problem indeed, I still want to minimize the costs. Therefore, I will search if I can replace Azure Even Hub (as a destination for azure Diagnostic Logs) with: 1- a Kafka instance that is not connected to an Azure event hub. 2- or by archiving in storage accounts (most cost-effective approach) and find a way to extract the log entries from there and ingest them to Loki.

Azure diagnostic settings allow sending to a partner solution, but I have not tried it out yet.

Edit: A more cost-effective approach: The storage account blob container can get mounted to a directory of a second Promtail instance (as suggested here).

On promtail.yaml config yaml, add a new job with its static_configs section of the "scrape_configs" directing Promtail to read from the mounted directory (e.g., /var/log) as follows:

scrape_configs:
- job_name: system
  static_configs:
    - targets:
        - localhost
      labels:
              job: varlog
              __path__: /var/log/**/*.log

Unlike the first approach which streams live log entries directly from azure diagnostic logs using an event hub, the second approach reads archived files that already got written to cold storage (indirectly - even the logs got written to azure storage accounts a while ago).

Naveen Sharma · Accepted Answer · 2024-03-01 13:03:21Z

1

steps to Ingesting logs from Azure Blob Storage to Loki

create a Event Hubs Namespace and Event Hub In azure portal go the Azure Blob Storage and select Events on the left menu, and then select + Event Subscription on the toolbar.
Enter a name for the event subscription.
Enter a name for the system topic. A system topic provides an endpoint for the sender to send events. For more information, see System topics
select the end point type as Event hub

enter image description here

Event subscription triggers when actions, such as those involving blobs, file shares, queues, tables, and any action with their respective blobs, occur and sends them to the Event Hub.

enter image description here

Please refer to this link for Azure Event Hubs to Loki and link1 to read logs from Azure Storage Accounts into Loki.

Steps to connect even hub to loki:

Go the event hub and next, decide on the authentication method you'll use: either OAuth or connection string. If using OAuth, make sure you have the required credentials set up. If using a connection string, obtain it from Azure.
In your Loki configuration file, set up the loki.source.azure_event_hubs component. Specify the Event Hub's namespace, list the Event Hubs you want to consume, and define where the logs will be forwarded.
Configure the authentication block with your chosen method and credentials. You can also customize other settings like group ID, relabeling rules, and whether to use incoming timestamps.
Ensure the destination specified in forward_to is correctly configured to receive logs. This destination could be an instance of LogsReceiver.

answered Mar 1, 2024 at 13:03

Naveen Sharma

1,5541 gold badge6 silver badges8 bronze badges

2 Comments

kloudkid Over a year ago

Thanks a lot for the answer! I understood I cannot use the basic pricing for event hubs. I configured the first part (system topic, event hub, etc). For the second part, I found very few example configurations online (like github.com/grafana/loki/issues/9562), which does not seem to work. I have a Loki Distributed deployment, and use connection_string for the authentication mechanism. Can you refer me to working Promtail example config? Do I have to define the LogReceiver (e.g., "loki.write") in the same config file?

kloudkid Over a year ago

I think I found out why I did not need the forward_to / logReceivers. I will add a detailed comment/answer how I configured it. Thanks again Naveen

Collectives™ on Stack Overflow

Ingesting logs from Azure Blob Storage to Loki

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related