How to filter out LogRecords in a processor , when using OpenTelemetry towards Azure Monitor?

Question

I am using OpenTelemetry and have it linked up with Azure Monitor (App Insights). I see all structured logging and traces appear in AI, which is nice.

I have read about the difference in tracing vs logging and that they are different pipelines. However, from an observability point of view, I need both in my (distributed application).

Ideally, I want to add a lot of logging, using ILogger, defining the LogLevel correctly. But then I want to make sure that at 'configuration time', so without code changes, the output level towards Application Insights can be adapted, in case issues occur, for example.

That's why I created a BaseProcessor<LogRecord> that I have wired up in the Application startup, with the following code.

// AIN : Using span processors to filter out telemetry
// This is used for filtering traces (dependencies, etc), which are called activities
builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) =>
    builder.AddProcessor(new AzureMonitorActivityFilterProcessor()));
// Here we are filtering the logs with a Processor
builder.Services.ConfigureOpenTelemetryLoggerProvider((sp, builder) =>
    builder.AddProcessor(new AzureMonitorLogFilterProcessor()));

builder.Services.AddOpenTelemetry()
    .WithLogging()
    .WithTracing()
    .WithMetrics()
    .UseAzureMonitor();

And in the next part, I have the code (just empty, ready to implement) for the BaseProcessor. The OnStart and OnEnd methods are being called, and I was assuming I would be able to indicate on a certain LogRecord (based on external configuration) that is should not be forwarded/sent in the pipeline towards the exporter.
But I don't see how I can do that.

using System.Diagnostics;
using Microsoft.Extensions.Logging;
using OpenTelemetry;
using OpenTelemetry.Logs;


public class AzureMonitorLogFilterProcessor : BaseProcessor<LogRecord>
{
    // The OnStart method is called when an activity is started. This is the ideal place to filter activities.
    public override void OnStart(LogRecord log)
    {
        if(log.LogLevel <= LogLevel.Information)
        {
            // What to set here, so that the log is not sent to the Azure Monitor?
        }
    }

    public override void OnEnd(LogRecord log)
    {
        if(log.LogLevel <= LogLevel.Information)
        {
            // What to set here, so that the log is not sent to the Azure Monitor?
        }
        base.OnEnd(log);
    }
}

Maybe I am totally wrong and using Processors for things they are not intended for, but I'd be happy to be pointed to the right way of doing this.

Thanks

BaseProcessor<LogRecord> in OpenTelemetry does not support stopping logs from being exported, filtering must be done at the exporter level instead. — Pavan
– Pavan, Commented May 15 at 8:40
@Pavan thanks for the reply. is there a way to inject custom logic at that exporter level? — Sam Vanhoutte
– Sam Vanhoutte, Commented May 16 at 5:19
Yes, you can inject custom logic at the exporter level by wrapping the actual exporter (e.g., AzureMonitorLogExporter) in a custom class. This allows you to apply filtering or transformation logic before the telemetry is sent. OpenTelemetry doesn’t support dropping logs inside BaseProcessor<LogRecord>, so exporter-level filtering is the correct approach. — Pavan
– Pavan, Commented May 16 at 9:25

Pavan · Accepted Answer · 2025-05-19 12:18:55Z

0

How to filter out LogRecords in a processor , when using OpenTelemetry towards Azure Monitor?

When using ILogger with OpenTelemetry to forward logs to Azure Monitor (Application Insights), the BaseProcessor does not support dropping logs. Need to filter logs before they are sent to Azure Monitor, you need to wrap the exporter and apply filtering logic.

Install the below packages with the supported versions and add into the project file.

dotnet add package Azure.Monitor.OpenTelemetry.Exporter
dotnet add package OpenTelemetry.Extensions.Hosting

enter image description here

custom wrapper around the Azure Monitor log exporter:

public class FilteringLogExporter : BaseExporter<LogRecord>
{
    private readonly AzureMonitorLogExporter innerExporter;
    private readonly Func<LogRecord, bool> filter;

    public FilteringLogExporter(AzureMonitorLogExporter innerExporter, Func<LogRecord, bool> filter)
    {
        this.innerExporter = innerExporter;
        this.filter = filter;
    }

    public override ExportResult Export(in Batch<LogRecord> batch)
    {
        var filtered = batch.Where(filter).ToList();
        if (filtered.Count == 0) return ExportResult.Success;

        // Export filtered logs
        return innerExporter.Export(new Batch<LogRecord>(filtered, filtered.Count));
    }
}

OpenTelemetry logging Configuration:

builder.Logging.AddOpenTelemetry(logging =>
{
    var connectionString = builder.Configuration["AzureMonitor:ConnectionString"];

    var azureExporter = new AzureMonitorLogExporter(new AzureMonitorLogExporterOptions
    {
        ConnectionString = connectionString
    });

    var filteringExporter = new FilteringLogExporter(
        azureExporter,
        logRecord => logRecord.LogLevel >= LogLevel.Warning // Only send Warning and above
    );

    logging.AddProcessor(new SimpleLogRecordExportProcessor(filteringExporter));
});

Need to add Application insights connection string in local.settings.json correctly.

{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "Application insights connection-string"
  }
}

By using this process can be filter the logs which sent to the application insights.

answered May 19 at 12:18

Pavan

1,4872 gold badges4 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sam Vanhoutte May 20 at 11:05

hello Pavan and thanks for your help. However, it seems the AzureMonitorLogExporter is an internal/sealed class, so I can't use and reference it from my own wrapper, no?

Pavan May 20 at 11:12

Can you check by using the OpenTelemetryLoggerOptions.Filter instead of AzureMonitorLogExporter to filter logs before they’re sent to Azure Monitor.

cijothomas Jun 12 at 17:26

This cannot be done unless the exporter itself is public. Only option today is to filter in ILogger itself, but it does not allow filtering based on the actual log contents. github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/…

Collectives™ on Stack Overflow

How to filter out LogRecords in a processor , when using OpenTelemetry towards Azure Monitor?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related