Strip away the marketing and an "agent" is a short loop: give a language model a set of tools (functions it can call — search, a database query, a shell command), let it decide which to call and with what arguments, run the tool, feed the result back, and repeat until the task is done. That's it. The founding pattern in the literature is ReAct — Yao et al. showed that interleaving explicit reasoning traces with actions lets a model plan, act, observe the result, and adjust, which beats reasoning or acting alone. Schick et al.'s Toolformer showed a model could even teach itself when to call an API from a handful of examples. Those papers make the loop work. Keeping it working under real traffic is a different discipline.
A note on sources: the reliability of production agents is still mostly documented in engineering practice, not peer-reviewed papers — the academic literature is thin and moves slower than the field. Below I cite the primary research that does exist plus one reputable engineering guide, and flag the rest as craft.
How agents fall over
The failure modes are specific and recurring:
- Hallucinated tool calls. The model invents a tool that doesn't exist, or passes arguments in the wrong shape. If your loop doesn't validate the call against the tool's schema before executing, you get a crash — or worse, a silently wrong action.
- Infinite loops. The model calls a tool, doesn't like the result, calls it again, forever. Without a hard step limit and loop detection, one stuck request burns tokens and money until something times out.
- Unsafe or irreversible actions. The model decides the fix is to delete the table, send the email, or refund the charge. It's often right. When it's wrong, there is no undo.
- Compounding error. Each step is 95% reliable; ten steps in sequence is 0.9510 ≈ 60%. Long autonomous chains multiply their own mistakes — a point Anthropic's engineering guide makes explicitly.
Reliability is engineering, not a smarter model
None of these are fixed by a better model — a smarter model just fails less often, which hides the problem until scale exposes it. They're fixed by building around the model:
- Tool design. Tools should be hard to misuse: tight schemas, validated arguments, clear errors the model can recover from. A well-designed tool surface prevents whole classes of failure before they start.
- Guardrails. Screen inputs and outputs — Anthropic notes that using a second model to check a first outperforms one model doing both. Constrain what tools are even reachable in a given context.
- Retries with backoff. Tools fail transiently. Catch, retry, and feed a clean error back to the model rather than crashing the loop.
- Evals. You cannot improve what you don't measure. A test suite of real tasks with graded outcomes is the only way to know a prompt change helped instead of quietly regressing.
- Human-in-the-loop for irreversible actions. The single highest-leverage guardrail: pause and ask a person before anything you can't undo. Anthropic's guidance is to have agents "pause for human feedback at checkpoints." Reversible actions run free; irreversible ones stop for a human.
for step in range(MAX_STEPS): # 1. hard loop limit
call = model.next_action(history)
if not schema_valid(call): # 2. reject hallucinated calls
history.append(error("bad tool call")); continue
if call.tool in IRREVERSIBLE: # 3. gate the dangerous ones
if not human_approves(call): break
try:
result = run_with_retry(call) # 4. retry transient failures
except ToolError as e:
result = error(str(e)) # 5. recover, don't crash
history.append(result)
if call.tool == "finish": break
Why it matters: the same instinct that keeps a supercomputing job from corrupting a shared filesystem — validate inputs, bound the blast radius, gate the irreversible, measure everything — is exactly what keeps an agent from taking a bad action ten thousand times an hour. The model is the easy part. The reliability is infrastructure, and it's the part that decides whether your agent ships or gets pulled after the first incident.