Evaluating agents before letting them loose

Quick answer:

A demo proves the system can work. An evaluation proves how often it does. Two different questions, and only the second is any use for deciding.

Most agents reach production validated against a handful of examples chosen by whoever built them. That is understandable and insufficient: those examples are precisely the cases the system handles well, because they are the ones used to tune it.

What an evaluation set is

It is not a list of questions. It is a set of real cases where, for each one, the following is written down:

• The request, as a real user would phrase it, ambiguity included.
• The expected answer or action, agreed with somebody who knows the domain.
• The evidence it should use: which source holds the right answer.
• What it must not do: actions forbidden in that context.

That last point is the one almost nobody writes and the one that prevents most trouble. An agent that answers well but executes an action it had no business executing has failed, however impeccable the text.

Cases come from real history: tickets, emails, queries. Between fifty and two hundred is enough to start, and they must include the rare cases, not just the frequent ones.

The metrics that matter

Four, and none of them is plain accuracy:

1. Correct answer with correct evidence. A good answer is not enough: it has to rest on the right source. Being right by accident is a failure that has not surfaced yet.
2. Action correctness. What percentage of executed actions were the right ones, and how many were executed that should not have been.
3. Cost per case, which decides viability as much as accuracy does.
4. Escalation rate. How often the agent recognises it cannot proceed and hands over. An agent that never escalates is not safe, it is reckless.

Adversarial simulation

Before granting autonomy, you have to try to break it deliberately:

• Prompt injection in the content the agent retrieves: a document carrying an instruction disguised as text.
• Contradictory data across two trusted sources, to see whether it spots the conflict or picks at random.
• Out-of-scope requests, to check that it refuses rather than improvises.
• Failing tools returning errors or nonsense, to see whether it notices or builds on top of them.

This is not a security audit at the end of the project: it is part of development, and it is best done by somebody other than whoever built the agent.

The threshold and the signature

Evaluation is only useful if somebody writes down, before running it, what result counts as acceptable. For example: correct-answer-with-evidence above ninety per cent, zero improper actions, cost per case below the viability threshold.

And somebody signs. Not the technical team alone: the agent's owner, the same person who answers for its actions. Without that signature, the decision to release is taken out of exhaustion, which is how most of them are taken.

It is the same principle we apply in MVP60™: acceptance criteria are agreed at the start, not negotiated at the end when everyone is in a hurry.

After deployment

The evaluation set is not archived: it is run periodically. Models change, sources change and users ask new things. An agent that passed in March may fail in September without anybody touching a line of code.

Every real incident also becomes a new case in the set. That is how you avoid fixing one specific failure and breaking it again three months later.

How Galde can help

We build the evaluation set with the client's domain team, not in a lab: the cases come from their own history and the thresholds are signed by their owner. We leave it automated so it runs on every change, like any other regression test. It is how we work in our AI with Palantir AIP line.

Conclusion

Granting an agent autonomy without evaluation is a decision taken without data, however many meetings are held to dress it up. Building the set costs a well-spent week and serves for the whole life of the system.

What you cannot do is replace it with trust. Trust is the output of evaluation, not a substitute for it.