Evaluating Agent Performance

When you write a normal JavaScript function, you write a Unit Test. You pass in 2 + 2, and assert that the output is exactly 4.

You cannot write standard unit tests for LLMs. If you ask Claude, "Write a polite greeting," it might say "Hello" today, and "Greetings" tomorrow. Because LLMs are non-deterministic (probabilistic), standard equality tests (===) will fail.

So, how do you know if the Agent you built is actually good? You need an Evals (Evaluations) pipeline.

LLM-as-a-Judge

The industry standard way to test an AI is to use a second AI as a judge.

Imagine you built an AI customer support Agent. You want to test if it handles angry customers well.

1. Create a Dataset (The Test Cases) You create an array of fake user inputs in your test file:

• "My package is late, I hate this company!"
• "Where is my order?"
• "I want a refund NOW."

2. Run the Agent You loop through your array, feeding each fake input to your Agent, and save the Agent's responses.

3. The Judge You write an evaluation script. This script calls Anthropic using the most powerful model (Opus). You give Opus a rubric, and ask it to grade your Agent's responses!

async function evaluateResponse(userInput, agentResponse) {
  const prompt = `
    You are an expert QA grader. Review the interaction below.
    
    User Input: ${userInput}
    Agent Response: ${agentResponse}
    
    Rubric:
    1. Did the Agent remain polite? (Pass/Fail)
    2. Did the Agent avoid offering a refund autonomously? (Pass/Fail)
    
    Output a JSON object: { "polite": boolean, "refund_safe": boolean, "reasoning": "..." }
  `;

  // Call Claude Opus to judge the response
  const eval = await anthropic.messages.create({ ... });
  return JSON.parse(eval.content[0].text);
}

Automating the Evals

If you have 100 test cases, you can run this script automatically every time you change your Agent's System Prompt.

If your Agent scores 98% "Pass" on politeness today, but you tweak the system prompt tomorrow and the score drops to 60%, you instantly know your tweak broke the Agent.

Building an Evals pipeline requires time, but it is the only way to confidently deploy AI applications to production. Without evals, you are just guessing that your prompts work.