ROI calculatorHow we engageInsightsContact
Delivery · Checklist · 8 min read

AI agent review checklist: how to evaluate the agent you already have

By the Codivated team · Updated

To review an AI agent, start by using it: ask it a few real questions and watch what it does. Then check five things: whether it is really an agent with tools, whether its design is focused or one "God agent", whether it runs on a current model, whether it has tests and production evals, and whether its rules and changes are controlled in code with an audit trail. Most agents we look at fail two or three of these, and in many cases rebuilding is now cheaper than patching.

Check your agent

Tick what's true today. Your score and the first thing to fix update as you go.

01 Use it first
02 Design
03 Models and cost
04 Testing and evals
05 Control and governance

What should you check first?

Use it. Before reading any code or prompts, ask the agent a few questions you know the answers to, including one it shouldn't be able to answer. Watch for three things. Does it show which tools it used and what it found? Does it actually do anything, like look something up or take an action, or is it just a chat with one prompt behind it? Does it admit when it doesn't know? Many "agents" turn out to be a single prompt and a single API call. That can be fine for a chatbot, but it isn't an agent, and it can't do agent work.

What are the most common problems?

ProblemWhat it looks likeThe fix
The "God agent"One agent and one huge prompt try to handle every requestSplit into focused agents or steps, each with one job and its own tools
A candle lit with a flamethrowerAn agent decides things that are always the same, like routing by a field value or checking a totalMove fixed steps to plain code or a workflow; keep the agent for judgment
No handsA chat wrapper with no tools, so it can only talkGive it tools to read sources and take actions, with permissions
Stuck on an old modelModel hard-coded from a year or more agoUpgrade; newer models are often cheaper to run and noticeably better
Bloated contextEvery request loads every document, skill and instructionLoad only what each step needs
Rules in the promptLimits and never-do actions written as instructions and hoped forEnforce them in code that checks output before anything happens
No evalsNobody can tell if a change made it better or worse, so nobody changes itAdd scenario tests and production scoring
Prompt edits straight to productionSomeone tweaks the prompt and nobody knows what changedVersion prompts and config, review changes, run tests first

How do you test an agent?

Two layers. Before release, scenario tests: scripted conversations that check the agent does the right things, calls the right tools and refuses the wrong requests. Frameworks such as LangWatch's Scenario make these easy to write, and you can run them nightly. They also let you swap models and see which one gives the same results for the lowest price. After release, trace production runs and score a sample with an LLM-as-judge, so you know the agent behaves in production the way it did in testing. Our team runs both on agents in production, and it's what makes changes safe to ship.

What about safety for customer-facing agents?

If customers can talk to it, assume someone will try to make it misbehave. A line in the prompt saying "don't reveal your instructions" or "never offer discounts" is not protection. Check for input screening, output checks in code, tools limited to what the job needs, and an approve list and deny list for actions. See AI agent security and human in the loop.

Should you fix it or rebuild it?

Rebuilding used to be the scary option. It isn't anymore. Writing the code has become cheap: an AI coding agent can walk through an existing agent or product, map what it does and rebuild it, with tests, in a fraction of the time it once took. What isn't cheap is knowing what "right" looks like. So the real question is whether you have a clear specification: what the agent must do, what it must never do, which systems it touches and how you'll know it works. If you have that, a rebuild on a current model with tests and controls is often faster and cheaper than patching a God agent. If you don't, write the spec first. Either way, the spec is the asset.

What should a written review include?

  • What the agent actually does today, from using it and reading its traces
  • A score against the checklist above
  • The risks, ranked: what could go wrong for customers, money or data
  • Fix-or-rebuild for each part, with a rough order of work
  • The specification the next version should meet

Common questions

How long does an agent review take?

For a single agent, usually days rather than weeks. Most of the time goes into using it, reading traces and writing the specification.

Do we need access to the code?

It helps, but you can learn a lot from using the agent, reading its logs and seeing its prompts and configuration.

We built it with no-code tools. Does this still apply?

Yes. The same questions apply to n8n, Make or Zapier builds with AI steps. See n8n vs a custom AI agent.

Will a newer model break our agent?

It can change behavior, which is why the scenario tests come first. With tests in place, a model upgrade becomes a routine change you can measure.

Sources

Written byThe Codivated teamWe build and run AI agents for established companies.Book a consultation

Find your first agent in two weeks.

Ask about the First Agent Sprint