AWS Bedrock: cannot invoke Anthropic Claude Sonnet 3.5 v2 model, raises error "Invocation of model ID with on-demand throughput isn’t supported."

Question

I am trying to invoke the Anthropic Claude Sonnet 3.5 v2 model in AWS Bedrock. Here is my code (in Python using Boto3):

import boto3
import json

bedrock_runtime = boto3.client(service_name="bedrock-runtime")

body = json.dumps({
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Hello, world"}],
    "anthropic_version": "bedrock-2023-05-31"
})

response = bedrock_runtime.invoke_model(body=body, modelId="anthropic.claude-3-5-sonnet-20241022-v2:0")

print(response.get("body").read())

I get the error:

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.

What does this mean, and how do I invoke the model through an inference profile?

Asfand Qazi · Accepted Answer · 2025-04-14 11:51:29Z

As the error says, you must provide the ID of an inference profile and not the model for this particular model. The easiest way to do this is to provide the ID of a system-defined inference profile for this model. You can find it by invoking this awscli command with the correct credentials defined in the environment (or set via standard flags):

aws bedrock list-inference-profiles

You will see this one in the JSON list:

{
  "inferenceProfileName": "US Anthropic Claude 3.5 Sonnet v2",
  "description": "Routes requests to Anthropic Claude 3.5 Sonnet v2 in us-west-2, us-east-1 and us-east-2.",
  "inferenceProfileArn": "arn:aws:bedrock:us-east-1:381492273274:inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0",
  "models": [
    {
      "modelArn": "arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
    },
    {
      "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
    },
    {
      "modelArn": "arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
    }
  ],
  "inferenceProfileId": "us.anthropic.claude-3-5-sonnet-20241022-v2:0",
  "status": "ACTIVE",
  "type": "SYSTEM_DEFINED"
}

Modify the invoke_model line in your code to specify the ID or ARN of the inference profile instead:

response = bedrock_runtime.invoke_model(
  body=body,
  modelId="us.anthropic.claude-3-5-sonnet-20241022-v2:0",
)

Ashish Chavan · Accepted Answer · 2025-03-10 17:00:24Z

0

You can add the ARN of an Inference profile to your modelID itself while invoking the model.

response = bedrock_client.invoke_model(

       modelId="arn:aws:bedrock:us-east-1::model/your-bedrock-model-arn"  

       prompt="Your prompt here"

   )

answered Mar 10 at 17:00

Ashish Chavan

1

Comments

Rasheed · Accepted Answer · 2025-04-08 10:46:47Z

Firstly ensure you have requested access to the model you'd like to use through the console.
Run aws bedrock list-inference-profiles on awscli

set model id to the inferenceProfileArn

"inferenceProfileSummaries": [
        {
            "inferenceProfileName": "EU Anthropic Claude 3 Sonnet",
            "description": "Routes requests to Anthropic Claude 3 Sonnet in eu-central-1, eu-west-1 and eu-west-3.",
            "createdAt": "2024-08-26T00:00:00+00:00",
            "updatedAt": "2024-08-26T00:00:00+00:00",
            "inferenceProfileArn": "arn:aws:bedrock:eu-west-1:374706529895:inference-profile/eu.anthropic.claude-3-sonnet-20240229-v1:0",
            "models": [
                {
                    "modelArn": "arn:aws:bedrock:eu-west-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
                },
                {
                    "modelArn": "arn:aws:bedrock:eu-west-3::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
                },
                {
                    "modelArn": "arn:aws:bedrock:eu-central-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
                }
            ],
            "inferenceProfileId": "eu.anthropic.claude-3-sonnet-20240229-v1:0",
            "status": "ACTIVE",
            "type": "SYSTEM_DEFINED"

Collectives™ on Stack Overflow

AWS Bedrock: cannot invoke Anthropic Claude Sonnet 3.5 v2 model, raises error "Invocation of model ID with on-demand throughput isn’t supported."

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related