Using Images with the API (Vision)

We explored Claude Vision in the web interface, but using it via the API unlocks massive automation potential. Imagine a script that automatically processes hundreds of scanned receipts and outputs the total cost to a database.

Base64 Encoding

The biggest difference between the Web UI and the API is how you hand the image to Claude. You can't just attach a file pointer. You have to convert the actual image into a text format that the API can read.

This format is called Base64.

Base64 is a way of translating binary data (like an image file) into a long, seemingly random string of text (e.g., iVBORw0KGgoAAAANSUhEUgAA...).

In Node.js, you can read a file from your hard drive and convert it to Base64 in two lines of code:

const fs = require('fs');

// Read the image file from your computer
const imageBuffer = fs.readFileSync('./my-receipt.png');
// Convert it to a base64 string
const base64Image = imageBuffer.toString('base64');

Sending the Image in the Payload

Once you have the Base64 string, you include it in the messages array. However, because you are sending both text and an image in the same message, the content property changes from a simple string into an array of objects.

const response = await anthropic.messages.create({
  model: 'claude-3-haiku-20240307',
  max_tokens: 500,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image',
          source: {
            type: 'base64',
            media_type: 'image/png', // Must match the file type
            data: base64Image, // Your long string goes here
          },
        },
        {
          type: 'text',
          text: 'What is the total amount listed on this receipt?'
        }
      ]
    }
  ]
});

Cost and Limitations

Processing images is computationally heavy. Claude calculates the token cost of an image based on its physical dimensions.

If you upload a massive 4K photograph just to ask what color a shirt is, you are wasting money. For most API use cases (like OCR on documents or receipts), you should resize and compress the image before sending it to Anthropic.

Additionally, just like with text, the image is stateless. If you upload the image in Message 1, and then ask a follow-up question about the image in Message 3, that Base64 string must still be in the chatHistory array you send back, meaning you pay to process the image again.