Intermittent Issue with App Insights not logging Custom Dimensions

We have some wrappers written around the ILogger that translate name-value pairs into custom dimensions. We've been using this code for years and have struggled with intermittent issues where sometimes we get customDimensions and other times we do not. This gets pretty frustrating when trying to triage errors as you can imagine. Maybe I have a bug in my implementation that I'm just not seeing.

This particular case: The application is a .net Core 9 Web API. It makes an http call to another service (a durable function) and is supposed to log the response, which is json with all the status urls - the same response you get if you use the runner in Azure portal.

The issue is that in this one case, App Insights just stops logging any custom dimensions in this one class where other places in the same API code function normally and write customDimensions.

For configuration, we have something like this:

where the code in question is in the namespace MyApi.Api.Application.Common.Services

    {
      "Logging": {
        "LogLevel": {
          "Default": "Debug",
          "Microsoft": "Error",
          "Microsoft.AspNetCore": "Error",
          "Microsoft.Hosting.Lifetime": "Error",
          "Microsoft.EntityFrameworkCore": "Error",
          "MyApi.Api.Application.Common.Services": "Debug"
        },
        "ApplicationInsights": {
          "samplingSettings": {
            "isEnabled": false
          },
          "EnableLiveMetricsFilters": false,
          "LogLevel": {
            "Default": "Debug",
            "Microsoft": "Error",
            "Microsoft.AspNetCore": "Error",
            "Microsoft.Hosting.Lifetime": "Error",
            "Microsoft.EntityFrameworkCore": "Error",
            "MyApi.Api.Application.Common.Services": "Debug"
          }
        }
      },
      "AllowedHosts": "*"
    }

Yes, filtering and sampling are enabled because this particular app is the backbone of our system, so it is very, very chatty in the logs.

The app insights service registration looks like this:

    public static void RegisterLogging(this IServiceCollection services, IConfiguration configuration, IWebHostEnvironment environment)
    {
        services.AddSingleton(environment);

        if (!string.IsNullOrEmpty(configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
        {
            services.AddLogging(builder =>
            {
                builder.AddApplicationInsights();
            });

            services.TryAddScoped<TelemetryClient>();

            services.AddApplicationInsightsTelemetry(options =>
            {
                options.ConnectionString = configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
            });
        }
    }

We check for existence of the connection string because it is auto-configured into our app services in Azure, but not always enabled on the developer's machine. app insights would throw an error if you called configuration with a non-existent connection string config value.

We have some extension methods that we use to abstract away some of the logging intricacies, so I'm going to show an example of the pathway used by one of our most common implementations.

    public static ILogger WriteDebug(this ILogger logger, string message,
        params (string, object)[] properties)
    {
        logger.WriteDebug(message, properties.ToLoggingScope(), null);

        return logger;
    }

    public static void WriteDebug(this ILogger logger, string message, IDictionary<string, object> properties = null, params object[] arguments)
    {
        properties = properties ?? CreateDefaultScope();

        using (logger.BeginScope(properties))
        {
            logger.LogDebug(message, arguments);
        }
    }

    public static IDictionary<string, object> ToLoggingScope(this (string, object)[] properties)
    {
        return properties
            .GroupBy(prop => prop.Item1, StringComparer.OrdinalIgnoreCase)
            .Select(grp => new KeyValuePair<string, object>(grp.Key, grp.LastOrDefault().Item2))
            .ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
    }

    private static IDictionary<string, object> CreateDefaultScope()
    {
        return new Dictionary<string, object>()
        {
            ["ScopeId"] = Guid.NewGuid().ToString("N")
        };
    }

And usage most often looks like this:

public class MyService(ILoggerFactory loggerFactory)
{
    private readonly ILogger<MyService> = loggerFactory.CreateLogger<MyService>();

    public void DoSomething(int myId)
    {
        _logger.WriteDebug("Begin DoSomething", 
            (nameof(myId), myId));
    }
}

This is the part that is frustrating. Sometimes, logged records will have customDimensions and other times not.