1

I'm trying to figure out how to add middleware to a chat client added to the Kernel. It seems like I need the actual instance of the chat client in order to add the middleware?

This is the documentation for the chat client middleware https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai#custom-ichatclient-middleware

This is my setup

services.AddKeyedTransient<Kernel>("Kernel", (sp, _) =>
{
    ...
    var bedrockRuntime = new AmazonBedrockRuntimeClient(awsCredentials, clientConfig);

    var kernelBuilder = Kernel.CreateBuilder();
    kernelBuilder.AddBedrockChatClient(
        modelId: awsBedrockOptions.GetModel(LlmOptions.ModelPurpose.General),
        bedrockRuntime: bedrockRuntime
    );

    return kernelBuilder.Build();
});

This is the middleware:

public sealed class LlmLoggerChatClient(
    IChatClient innerClient,
    LlmLogger llmLogger)
    : DelegatingChatClient(innerClient)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {

        var response = await base.GetResponseAsync(messages, options, cancellationToken);
        await llmLogger.LogAsync(...)
        return response;
    }

    public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        await foreach (var update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
        {
            yield return update;
        }
    }
}

And this is how I finally use the kernel


ChatCompletionAgent agent = new()
{
    Name = "Search Agent",
    Kernel = kernel, // Injected via DI
    ...
};

var response = await agent.InvokeAsync(...

At what point should I/can I extend the Bedrock chat client?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.