Can't send image to Bedrock using ConverseCommand in JavaScript

Question

I'm trying to upload an image to AWS Bedrock as part of a conversation using the ConverseCommand (in @aws-sdk/client-bedrock-runtime v3.686). According to the documentation, it requires the bytes as a Uint8Array, which I think is the same as a buffer. However, I have tried many permutations and combinations with .buffer and new TextEncoder().encode(imageBuffer) and it always says something like:

I'm sorry, but you haven't actually shared an image with me yet. Could you please upload an image and I'll take a look at it?

It feels like this should be trivial, but I haven't been able to find any working code using the @aws-sdk/client-bedrock-runtime that includes an image or document as part of the messages. Any ideas?

const fs = require("fs");
const {
  BedrockRuntimeClient,
  ConverseCommand,
} = require("@aws-sdk/client-bedrock-runtime");

const modelId = "anthropic.claude-3-haiku-20240307-v1:0";

let conversation = [];

const askQuestion = async () => {
  const client = new BedrockRuntimeClient({
    region: "us-east-1",
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    },
  });

  let content = {};
  const imageBuffer = fs.readFileSync("./butterfly.png");

  content.image = {
    format: "png",
    source: {
      bytes: imageBuffer.buffer,
    },
  };
  content.text = "What's in this image?";
  conversation.push({
    role: "user",
    content: [content],
  });

  try {
    const response = await client.send(
      new ConverseCommand({
        modelId,
        messages: conversation,
      })
    );
    conversation.push(response.output.message);
    console.log(response.output.message.content[0].text);
    return response.output.message.content[0].text;
  } catch (err) {
    console.log(`ERROR: Can't invoke '${modelId}'. Reason: ${err}`);
  }
};

askQuestion();

Answer from Oluwafemi Sule was correct, but I needed to change the way I sent the image as well. This worked:

const imageBuffer = new Uint8Array(fs.readFileSync("./butterfly.png").buffer);

  content.push({
    image: {
      format: "png",
      source: {
        bytes: imageBuffer,
      },
    },
  });
  content.push({ text: "What's in this image?" });
  conversation.push({
    role: "user",
    content,
  });

  try {
    const response = await client.send(
      new ConverseCommand({
        modelId,
        messages: conversation,
      })
    );
    conversation.push(response.output.message);
    console.log(response.output.message.content[0].text);
    return response.output.message.content[0].text;
  } catch (err) {
    console.log(`ERROR: Can't invoke '${modelId}'. Reason: ${err}`);
  }

I also found this example using C# and it feels like I am doing the same thing using JavaScript, but it isn't working: c-sharpcorner.com/article/… — Jeff Bonnes
– Jeff Bonnes, Commented Nov 11, 2024 at 10:11

Oluwafemi Sule · Accepted Answer · 2024-11-11 23:49:12Z

1

The document is handled as a text because a text field is provided. This is because the implementation prioritizes the text over the image.

The type definition for an image content block is such that only the image field can have a value. You should set only the image field for an image content block.

export interface ImageMember {
    text?: never;
    image: ImageBlock;
    document?: never;
    toolUse?: never;
    toolResult?: never;
    guardContent?: never;
    $unknown?: never;
  }

In order to ask about the content in the image, make another content block containing the question.

  const content1 = {
    text: "What's in this image?"
  };

  const content2 = {
    image: {
      format: "png",
      source: {
        bytes: imageBuffer.buffer,
      },
    }
  };

  conversation.push({
    role: "user",
    content: [content1, content2],
  });

edited Nov 11, 2024 at 23:49

answered Nov 11, 2024 at 23:42

Oluwafemi Sule

39.3k1 gold badge63 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jeff Bonnes Over a year ago

Thanks! I needed to change imageBuffer to const imageBuffer = new Uint8Array(fs.readFileSync("./butterfly.png").buffer) but then it worked!

Oluwafemi Sule Over a year ago

Glad you figured it out.

Collectives™ on Stack Overflow

Can't send image to Bedrock using ConverseCommand in JavaScript

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related