People often judge an AI agent by its best moment: the clever reply, the tricky document it read perfectly. A business should judge it by its worst ordinary day. If an agent handles a refund request correctly four times out of five, it is wrong one time in five, and nobody knows in advance which time that will be. For real work, consistency matters more than brilliance. It can be measured, and it can be improved.
Why agents are inconsistent in the first place
Language models produce their output by choosing likely next words, and there is usually some randomness in that choice. Ask the same question twice and the wording will differ. Most of the time the meaning stays the same. Sometimes it does not, especially on borderline cases where two answers seem almost equally reasonable.
Agents add more sources of variation. Each step depends on the one before. A small difference early on, such as reading a date slightly differently or choosing a different tool first, can lead to a completely different final result. Longer tasks with more steps are therefore less consistent than short ones.
Outside factors matter too. Model providers update their models. Search results change. A customer’s history grows. An agent that was consistent last month may not be this month, even if nobody touched it.
Measuring consistency in practice
The basic method is simple: run the same input through the agent several times and compare the results.
- Pick a set of real cases, say fifty, covering easy, normal and borderline work.
- Run each case five times without changing anything.
- For each case, check whether the key outcome is the same every time. The key outcome is the decision, not the wording: approved or escalated, which category, which amount.
- Count how many cases gave the same outcome on all five runs.
Say 44 of the 50 cases were stable, and 6 flipped at least once. Those six are the interesting ones. Look at them closely. Often they share a pattern: an unclear policy, an ambiguous kind of message, a field that is sometimes missing.
Track two numbers separately. The first is how often the agent is right. The second is how often it gives the same answer. An agent can be consistently wrong, which is actually easier to fix than being randomly right, because a consistent error points to a clear cause.
Ways to make an agent more consistent
Several practical changes reduce variation, usually without making the agent less capable.
- Lower the randomness setting. Most models have a setting, often called temperature, that controls how much variety they add. For decision tasks, keep it low.
- Ask for structured output. An agent that must choose from a fixed list of categories and fill defined fields varies less than one that writes a free-form answer.
- Move rules out of the prompt and into code. “Refunds above LKR 10,000 go to a manager” should be a line of code that checks the amount, not a sentence the model might weigh differently each time.
- Shorten the chain. Break a long task into smaller steps with checks between them, so an early wobble does not carry through.
- Fix the model version. Where the provider allows it, use a specific model version rather than whatever is newest, and test before upgrading.
- Clarify the policy. Many inconsistent cases are ones where the business itself has not decided the rule. The agent is showing you a gap in your own procedures.
When some variation is fine
Not every task needs identical output. A drafted reply to a customer can be worded differently each time as long as it is accurate and polite. A product description can vary in style. Demanding word-for-word sameness there wastes effort.
The line to draw is between the decision and the expression. Decisions should be stable: whether to approve, how to classify, what amount, who to route to. Expression can vary within limits. When you set up your measurements, be clear about which parts of the output are decisions and which are wording.
Also be honest about what consistency cannot give you. A perfectly consistent agent can still be wrong on a whole class of cases. Consistency testing tells you the agent is predictable. Accuracy testing, against answers a person has checked, tells you whether it is right. You need both.
Keep watching after launch
Run the same consistency check on a schedule, perhaps monthly, using the same fixed set of cases. If the stable share drops, something has changed: the model, the data or the inputs. Catching that early is far cheaper than finding out through complaints.
A good starting point is modest. Take ten cases your team handles every week, run them through the agent five times each, and see how many come back the same. That one afternoon will tell you more about whether the agent is ready than any demo.
