Questions to ask before you build an AI agent
By the Codivated team · Updated
Before you build an AI agent, answer four sets of questions. The job: can you describe it in a sentence, does it need judgment, and is there no product that does it already? The value: what number will it beat, and what does a mistake cost? The people: who owns it after launch, and who approves its actions? The data: can it reach the systems it needs, and do you have real cases to test it against? If you can't answer most of these yet, you're not ready to build, and that's the cheapest thing to find out.
Check your agent idea
Tick what's true today. Your score and the first thing to settle update as you go.
Why ask before you build?
Because the expensive mistakes happen before anyone writes code. When we reviewed recent job posts from companies hiring for AI agent work, 33 of the 47 that described the project (70%) never said who would own the system once it was live. Many described work that was really a fixed workflow. Neither problem shows up in a demo. Both show up a month after launch, when nobody is watching the agent and it's doing a job a cheaper tool could have done.
Is it really an agent job?
An agent earns its cost when the work needs judgment: reading a messy email, deciding what kind of request it is, checking several systems and choosing what to do. If the person doing the work today follows the same steps every time, build a workflow instead. If they read something first and then follow fixed steps, a workflow with one AI step usually fits. People often ask for an "agent" when an automation is the better answer. Take the agent, workflow or off-the-shelf quiz if you're unsure.
What number will it beat?
Pick one number the agent exists to move: minutes per order, time to first reply, invoices handled per person, errors per month. Measure it for two weeks before the build. Without a baseline, nobody can say whether the agent worked, and pilots without a clear number are the ones that stall. See how to measure AI ROI and why AI pilots fail.
Who owns it after launch?
Treat an agent like a new hire. It needs a manager who reviews its work, answers its questions and tunes it as the business changes. Name that person before the build starts. The person who does the work today should help shape it too: they know the exceptions nobody wrote down, and they're the ones who'll trust it or quietly work around it.
What must it never do?
Decide early which actions the agent can take on its own, which need a person's approval and which it may never take, like deleting records or sending money. Those rules belong in code, not just in the prompt. At first, a person should approve everything. See human in the loop AI agents.
Can it reach your systems and data?
List every system it must read from and write to. Check that each one has an API or another safe way in, and that the agent can have its own login with only the permissions it needs. Then gather 50 or so real past cases with the right answers. They become the test set that tells you whether it's ready, and they keep telling you every time you change it.
What should you have before the build starts?
- A one-page spec: the job, the inputs, the outputs, the rules and the never-do list
- The baseline number and the target
- A named owner and the approvers
- Access to the systems, with the agent's own logins
- A test set of real cases with known answers
- A plan to start in practice mode, with a person approving every action
Common questions
What if we can't answer most of these?
That's normal, and it's the point of asking. Start with the job and the number to beat. A short discovery, a few days of watching the work and looking at real cases, usually answers the rest.
Should we build our first agent or buy one?
Buy when a product does the job the way you need. Build when the work spans your own systems and rules. Many companies do both.
How small should the first agent be?
One job, one team, one number. A small agent that works and is trusted beats a broad one that nobody uses.
Who should be in the room for these questions?
The person who does the work today, their manager, someone who knows the systems, and whoever will approve the budget.
Sources
- Building effective agents, Anthropic: start simple, and use agents only where the task needs them.
- AI Risk Management Framework, NIST: defining roles, oversight and measurement before deployment.