Most founders who try to work out how to build an AI agent start in the wrong place: they open an agent builder, connect a model and a few apps, and only then ask what the agent should actually do. A week later they have a clever demo that nobody trusts with real work. This guide is the other way round. It is the build order we use on client projects, written so you can follow it yourself and end up with one agent doing one real job in your business.
To build an AI agent for your business, pick one repeated task with a clear finish line, collect real past examples with the right answers, write a one-page spec, give the agent only the tools that task needs, write one short system prompt, test it on 30 held-back cases, run it beside a person, then go live with daily review.
Disclosure: some tool links in this guide are referral links from our AI Stack page. If you sign up through them we may earn a commission at no extra cost to you. We list tools we use or have vetted, and you can run every step here with other tools.
If you are new to the idea, read what an AI agent is first; this post assumes you know the basics and want the do-it-yourself walkthrough. It is platform-neutral on purpose. When you want to see the same method applied to a specific channel, we have worked builds for AI voice agents, AI agents for lead generation and an AI appointment setter. The worked example below is a lead qualification agent, because it is the most common first agent we see in B2B service businesses.
What you will have at the end
- One chosen task, scored against your other candidates, with a written reason it goes first.
- An answer key: 50 real past examples of that task with the correct outcome, split into 20 for building and 30 held back for testing.
- A one-page agent spec: trigger, inputs, outputs, allowed actions, forbidden actions, when to hand over to a person and what "done" means.
- A tool permission table that says exactly what the agent can read, what it can change and what needs your approval.
- One short system prompt, the only prompt in this build.
- A working agent on the platform you choose, passed against your 30 held-back cases, piloted beside a person and live with a log you review every day.
What an AI agent is made of
Every agent we build, whatever the platform, comes down to the same seven parts. Knowing them stops you from buying a tool for a part you have not designed yet.
| Part | What it is | Where you define it in this guide |
|---|---|---|
| Trigger | The event that starts a run: a new form entry, a new row, an email, a schedule, a call | Step 3 (spec) and Step 7 (build) |
| Inputs and context | The data the agent receives and can look up for this run | Step 3 (spec) and Step 5 (tools) |
| Model | The large language model that reads the inputs and decides the next action | Step 7 (build) |
| Instructions | The system prompt: role, rules, output format, when to stop | Step 6 (prompt) |
| Tools | Actions the agent can take: search, read a page, write a row, draft an email, book a slot | Step 5 (tools and permissions) |
| Human checkpoint | The points where a person approves, corrects or takes over | Step 3 (spec) and Step 9 (pilot) |
| Log | A record of every run: input, decision, reason, action, result | Step 7 (build) and Step 10 (go live) |
Build an AI agent, buy one or hire someone?
Before you build an AI agent yourself, check that building is the right call. There are three routes and each one fits a different situation. Score your situation honestly against this table.
Asset 1: Build, buy or hire decision table| Factor | Buy a ready-made agent product | Build it yourself on an agent platform | Hire a developer or agency |
|---|---|---|---|
| Your task is standard (for example answering calls, sending email sequences) | Strong fit: products exist for common jobs | Possible, but you rebuild what already exists | Only if the product route failed |
| Your task depends on your own rules (how you qualify, price, route) | Weak fit unless the product lets you write those rules | Strong fit | Strong fit |
| Someone on the team can spend 5 to 10 focused hours a week for a month | Not needed | Needed | Not needed, but you still own the spec and the answer key |
| The agent will write to your CRM, send messages or touch money | Check what the product logs and lets you approve | You control every permission | Ask how permissions, approvals and logs are handled |
| You need it working this month and it is core to revenue | Fastest if a product fits | Slowest the first time | Fast if the provider has built this type before |
| You want to own and change it later | You own settings, not the system | You own everything | Insist it is built in your accounts with documentation |
Decision rules: if a ready-made product does the task and lets you write your own rules, buy it and use Steps 2, 8, 9 and 10 below to test it before you trust it. If the task depends on your own judgement and you have the hours, build it with this guide. If it is core to revenue and nobody has the hours, hire, and use this guide as your brief; our checklist on how to choose an AI automation agency covers what to ask.
One more check. Anthropic's engineering team, in its widely read guide to building effective agents, separates workflows (the model and tools follow a path you fixed in advance) from agents (the model decides its own next step), and recommends starting with the simplest solution that works. In our experience most first "agents" in a small business are really a workflow with one or two AI steps, and that is a good thing: it is cheaper, more predictable and easier to test. Step 4 helps you decide which one you are building.
Where AI agents for business work, and where they do not
| Works well | Works with a human checkpoint | Keep with a person |
|---|---|---|
| Sorting and tiering leads, tickets or applications against written criteria | Drafting first replies to new enquiries that a person approves | Negotiating price or terms |
| Researching a company or person from public pages and summarising it | Updating CRM fields from calls, emails or forms | Complaints, refunds and anything emotional |
| Extracting fields from documents into a sheet | Booking meetings into a calendar | Legal, tax, medical or financial advice |
| Answering routine questions from information you approved | Sending follow-ups on a schedule you set | Decisions you could not explain to the customer afterwards |
The limits are the same on every platform. An agent only knows what you give it, it can be confidently wrong, it can repeat a mistake across hundreds of records before anyone notices, and its running cost grows with every step and tool call. Each of those is handled by a specific step below: the answer key and test set (Steps 2 and 8), the permission table (Step 5), the pilot (Step 9) and the daily log review (Step 10).
Is this task a good first agent? Six questions
- Does the task happen at least weekly, ideally daily?
- Could you write down how a good person does it on one page?
- Do you have, or can you quickly collect, 50 past examples with the right outcome?
- Can the agent get every input it needs from systems you control or public pages?
- If the agent gets one case wrong, can a person catch and fix it before it hurts a customer?
- Can you measure success with a number (matches the right answer, time saved per item, replies, bookings)?
Decision rule: six yeses, start now. Four or five, start but fix the "no" answers first (usually question 3: collect the examples). Three or fewer, pick a different task; the scoring sheet in Step 1 will show you a better one.
What you need before you start
- An owner: one person who knows the task, writes the spec and reviews the log. This is not optional.
- Access to the systems involved (CRM, inbox, sheet, website) with the right to create an API key or connection, ideally a separate user or key for the agent so its actions are traceable.
- An account on a model provider and an automation or agent platform (see the tools table next). In our setup we connect our own model API key rather than paying per-task credits inside a platform, so costs stay visible.
- A shared sheet for the answer key, the test results and the run log.
- Two to four hours for Steps 1 to 3, a day for Steps 4 to 8, and one to two weeks for the pilot.
Tools that can run these steps (AI agent platforms and builders)
You do not need all of these. Pick one model and one platform; the rest are optional depending on the task. Descriptions come from our AI Stack page. Check each vendor's documentation for current features, limits and pricing before you commit.
| Step | Tool | How it fits |
|---|---|---|
| Model (the agent's reasoning) | Claude | The AI assistant we use daily to write, build and run our stack. Use it to draft the spec and prompt, and as the model inside the agent through its API. |
| Agent platform (no-code) | Make | The no-code automation backbone connecting most of our client tech stacks. Good when the agent is mostly a workflow with AI steps. |
| Agent platform (open source) | n8n | The open-source workflow engine we run alongside Make. Good when you want to self-host or need more control over each step. |
| Agent that works in your files and tools | Claude Code | The AI coding agent we run our GTM workflows through directly. Suits founders comfortable with a terminal who want the agent to run multi-step jobs. |
| Answer key, test sheet and run log | Google Workspace | Email, docs and file storage. A Google Sheet is all you need for the answer key, test results and log. |
| CRM the agent writes to | Pipedrive | A simple, visual CRM pipeline for teams that do not need Salesforce-level complexity. |
| Booking (if the agent books meetings) | Cal.com | Our preferred booking tool. It connects to Claude through MCP, so an AI teammate can check availability and book calls directly. |
| Voice channel (if the agent talks on the phone) | Vapi or Retell AI | The two voice agent platforms we build client agents on. The full voice build is in our voice agents guide. |
Zapier is another common no-code option; it is not on our AI Stack page, so we name it without a link. An AI agent builder is only as good as the spec and test set you give it, which is why the platform choice comes at Step 7, not Step 1.
How to build an AI agent, step by step
Work through these in order. Steps 1 to 6 are mostly writing and deciding; Steps 7 to 10 turn that into a working agent. Do not skip to Step 7.
Step 1: Pick one task with a clear finish line
Input: a list of repeated tasks from your own week and your team's. Do: write every candidate as one row in the sheet below and score it. Be specific: "qualify inbound leads from the website form" is a task, "sales" is not. Output: one chosen task with a written finish line, the state that tells you the agent is done (for example "every new lead has a tier, a reason and an owner in the CRM"). Check: two people on your team would describe the finish line the same way.
Asset 2: Task scoring sheetCOLUMNS (one row per candidate task)
task | how often (per week) | minutes per item | rules clarity (1-5) |
examples available (1-5) | inputs reachable (1-5) | damage if wrong (1-5, 5 = low damage) |
total = rules + examples + inputs + damage | finish line | owner
SCORING
rules clarity 5 = written rules exist 1 = "it depends, I just know"
examples 5 = 50+ with outcomes 1 = none recorded
inputs reachable 5 = all in our systems 1 = in people's heads
damage if wrong 5 = internal only, easy to fix 1 = customer, money or legal
Decision rules: rank by total, then by weekly volume times minutes per item. Never pick a task with "damage if wrong" of 1 as your first agent, however high it scores elsewhere. If two tasks tie, pick the one with more recorded examples.
Step 2: Build your answer key from real past examples
Input: 50 past instances of the task with what a good person decided. Do: put one instance per row with the inputs exactly as the agent would see them and the correct output. Where the decision needed judgement, write the reason in one sentence. Then split the sheet: 20 rows for building (you will use their patterns in the spec and prompt) and 30 rows held back for testing. Pick the 30 so they include easy cases, hard cases and cases where the agent should refuse or hand over. Output: a two-tab answer key, "build" and "test". Check: nobody writing the prompt looks at the test tab until Step 8.
This is the step that decides whether the agent will be any good. On lead qualification projects we ask the client's expert to walk us through how they tier companies, example by example, with the reason behind each call. We use the already-tiered rows to write the rules, then the expert and the agent each tier a fresh batch independently and we compare. Writing down the reasons is what turned "I just know" into rules an agent can follow. Anthropic's guide on agent evaluations makes a similar point: 20 to 50 tasks drawn from real cases are enough to start, and a good test case is one where two experts would independently reach the same verdict.
Asset 3: Answer key templateTAB "build" and TAB "test" (same columns)
case_id | input_1 | input_2 | input_3 | correct_output | reason (one sentence) |
case_type (easy / hard / should_hand_over / should_refuse)
RULES FOR THE TEST TAB
- 30 rows minimum
- at least 5 hard cases and at least 5 hand-over or refuse cases
- one correct_output per row; if experts disagree, fix the rule or drop the row
Step 3: Write the one-page agent spec
Input: your answer key's build tab and the finish line from Step 1. Do: fill in the template below in plain language. Write the rules from the reasons column in your answer key: every reason that repeats becomes a rule. List forbidden actions explicitly; an agent does not know what you would never do. Output: one page that a new hire could follow without asking you. Check: give the spec and five build rows (without answers) to a colleague; if they get at least four right, the spec is clear enough.
Asset 4: One-page agent specAGENT NAME: <SHORT_NAME>
OWNER: <PERSON_WHO_REVIEWS_THE_LOG>
GOAL (one sentence): <WHAT_THE_AGENT_ACHIEVES>
TRIGGER: <EVENT_THAT_STARTS_A_RUN>
INPUTS: <FIELDS_IT_RECEIVES> + <WHAT_IT_MAY_LOOK_UP>
OUTPUT FORMAT: <EXACT_FIELDS_AND_ALLOWED_VALUES>
RULES: 1. ... 2. ... 3. ... (from your answer key reasons)
ALLOWED ACTIONS: <LIST>
FORBIDDEN ACTIONS: <LIST, e.g. never email a contact, never delete a record>
HAND OVER TO A PERSON WHEN:
- inputs are missing or contradict each other
- confidence is low (the agent says "unsure")
- <TASK_SPECIFIC_TRIGGERS>
DONE MEANS: <FINISH_LINE_FROM_STEP_1>
MAX STEPS PER RUN: <NUMBER> (stop and hand over if reached)
LOG EVERY RUN TO: <SHEET_OR_TABLE>
Step 4: Decide workflow or agent, and draw the flow
Input: your spec. Do: list the steps a person takes, in order. If the order is always the same, you are building a workflow with AI steps: draw it as a straight line of boxes. If the person decides what to look up next based on what they found, you need an agent loop: draw the loop (look at inputs, choose a tool, read the result, decide whether to stop) and set the maximum number of steps from your spec. Output: a box diagram on paper or in a doc, each box labelled trigger, lookup, AI decision, check, action or log. Check: every box has an input and an output, and there is a box for "hand over to a person".
Decision rule: if you can draw it as a straight line, build the straight line. Add a loop only for the part that genuinely varies, such as "keep reading pages until you find what the company sells, up to three pages".
Step 5: Define tools and permissions
Input: your flow diagram. Do: list every action the agent takes and turn each one into a tool with a narrow purpose. Give each tool a plain name and a one-line description a new hire would understand, return only the fields the agent needs, and decide whether it reads or writes. Anthropic's article on writing tools for agents recommends a few well-chosen tools over many overlapping ones, clear unambiguous parameter names and helpful error messages; that matches what we see in practice. Start every write tool behind an approval step. Output: a filled permission table. Check: there is no tool the spec does not need, and nothing can delete, pay or send to a customer without a person approving it during the pilot.
Asset 5: Tool permission table| Tool name | What it does | Read or write | Approval during pilot | Approval after go-live | Limits |
|---|---|---|---|---|---|
| get_record | Fetches one CRM or sheet record by ID | Read | No | No | Only the fields listed in the spec |
| read_web_page | Returns the visible text of one public URL | Read | No | No | Max pages per run from the spec |
| update_record | Writes the agent's output fields to one record | Write | Yes | No, once accuracy targets are met | Only its own output fields, never contact details |
| draft_message | Creates a draft, never sends | Write | Yes | Yes, for the first month | Drafts only |
| hand_over | Assigns the case to a person with a reason | Write | No | No | Always available |
These names are examples. On a no-code platform each "tool" is usually a module or node, and on an agent framework it is a function you describe to the model; check your platform's documentation for how it defines tools.
Step 6: Write the system prompt
Input: your spec and permission table. Do: turn the spec into the short prompt below. This is the only prompt you need for the whole build: everything else is configuration, data and tests. Keep rules in the order a person applies them, put the output format last and include one or two worked cases from the build tab (never from the test tab). Output: a prompt saved in a doc with a version number and date. Check: read it aloud; every rule maps to a line in the spec and nothing in it contradicts the permission table.
Asset 6: System prompt (the only prompt in this build)You are <AGENT_NAME>, working for <COMPANY>. Your job: <GOAL_SENTENCE>.
For each case you receive <INPUT_FIELDS>. You may use these tools: <TOOL_LIST>.
Use the fewest tool calls needed. Stop after <MAX_STEPS> steps.
Rules, in order:
1. <RULE_1>
2. <RULE_2>
3. <RULE_3>
Never: <FORBIDDEN_ACTIONS>.
If inputs are missing, contradictory, or you are not confident, call hand_over
with a one-sentence reason instead of guessing.
Example: <ONE_BUILD_CASE_INPUT> -> <ITS_CORRECT_OUTPUT> because <REASON>.
Return only this JSON:
{"decision": "<ALLOWED_VALUES>", "reason": "<ONE_SENTENCE>",
"evidence": "<WHAT_YOU_SAW_AND_WHERE>", "confidence": "high|medium|low"}
Step 7: Build it on your platform
Input: flow diagram, permission table, prompt. Do: build the flow in this order, testing each piece with one build case before adding the next: (1) the trigger, fired manually at first; (2) the lookups that fetch inputs; (3) the model call with your prompt, your tools and structured output; (4) a validation check that rejects any output that is not valid JSON or uses a value outside your allowed list, and routes it to hand_over; (5) the action, set to "draft" or "approval needed"; (6) the log row. Use a separate API key or user for the agent, and connect your own model key where the platform allows it, so usage is visible. The exact node, module or setting names differ by platform: check your platform's documentation for each one rather than following screenshots from another tool. Output: a flow that takes one case from trigger to log. Check: run three build cases end to end; each produces a log row with input, decision, reason, evidence, confidence, action and timestamp.
Asset 7: Run log columnsrun_id | timestamp | case_id or record link | inputs (short) | decision | reason |
evidence | confidence | tool calls made | action taken | handed_over (yes/no) |
human_verdict (agree / disagree / fixed) | human_note | prompt_version | cost_estimate
Step 8: Test on your 30 held-back cases
Input: the test tab from Step 2. Do: run every test case through the agent and record its output next to the correct output. Grade the result, not the path: if the agent reached the right decision with a different sequence of tool calls, that is a pass. Read the full transcript of every failure and label it as a rule gap (fix the spec and prompt), a missing input (fix the lookup), a tool problem (fix the tool) or a bad test case (fix the answer key). Change one thing at a time, bump the prompt version and rerun all 30. Output: a scored test sheet and a short change log. Check: you meet the targets you set in the scorecard below, including the hand-over cases.
Asset 8: Test scorecard| Measure | How to calculate | Set your target before you run |
|---|---|---|
| Agreement | Cases where decision equals correct_output, divided by 30 | The level a careful new hire would reach on the same cases |
| Hand-over accuracy | Should-hand-over cases the agent handed over, divided by all such cases | All of them; a missed hand-over is the costly failure |
| Unnecessary hand-overs | Easy cases the agent handed over, divided by easy cases | Low enough that the person reviewing is not swamped |
| Format failures | Outputs rejected by your validation check | Zero after fixes |
| Steps per case | Average tool calls per case from the log | Within your max steps; falling, not rising, between versions |
Decision rule: do not move to the pilot until every should-hand-over case is handed over. A wrong tier on an easy case costs you a minute; a confident wrong answer on a case that needed a person costs you a customer.
Step 9: Pilot in shadow mode beside a person
Input: the tested agent and a person who normally does the task. Do: run the agent on real, live cases for one to two weeks while the person still does the task as normal. The agent writes to the log only. Each day the person fills in the human_verdict column. Before increasing volume, run the agent on a batch of 10 to 20 cases and work out the cost per case from your platform and model usage; this is how we estimate cost on every enrichment and agent project before scaling to hundreds of records. Output: a pilot log with a human verdict on every run and a cost per case. Check: live agreement is close to your test agreement; if it is much lower, your test set did not represent real cases, so add the failures to the test tab and go back to Step 8.
Step 10: Go live in stages, then review daily
Input: pilot results. Do: switch on real actions for the case types where the pilot showed the agent was reliable, and keep the others on draft or hand-over. Remove approval on a write tool only when its pilot record supports it. Review the log every day for the first two weeks, then weekly. After any prompt or tool change, rerun the 30 test cases before the change goes live, so a fix for one case does not break three others. Output: a live agent with a written list of what it does on its own and what still needs a person. Check: the owner can answer, from the log alone, how many cases the agent handled this week, how many it handed over and how many a person corrected.
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| Passes your demos, fails the test set | The prompt was tuned on cases you remembered, not on the answer key | Rebuild rules from the build tab reasons; test only on the held-back tab |
| Confident wrong answers | No permission to say "unsure"; the prompt rewards always deciding | Keep the hand_over instruction and the confidence field; route low confidence to a person |
| Hands over almost everything | Rules too vague, or the hand-over condition is too broad | Add the missing rules from failures; narrow the hand-over triggers to the spec list |
| Calls tools in long loops | No step limit, or a tool returns too much irrelevant text | Set max steps; make the tool return only needed fields; add an instruction to stop when the decision is clear |
| Output breaks the next step | Free text where the flow expects fixed fields | Ask for JSON with allowed values; validate and route failures to hand_over |
| Works for a week, then drifts | New kinds of cases, a changed website or CRM field, or an untested prompt change | Add new failures to the test tab; rerun all tests after every change; track prompt_version in the log |
| Cost per case higher than expected | Too many steps, long pages passed to the model, a larger model than needed | Trim tool output, cap pages, test a smaller model on the same 30 cases and compare |
A worked example (illustrative)
This example is fictional. The company, records and numbers are invented to show the inputs and outputs at each step; they are not client results.
The business: "Ledgerline", a fictional company that sells payroll software to accounting firms. Its founder spends several hours a week opening firm websites from a prospect list and deciding who is worth contacting.
Step 1, task: "Tier each firm on the prospect list as Priority 1, Priority 2 or Not a fit, with a reason, so outreach only goes to Priority 1 and 2." Finish line: every row has a tier, a one-sentence reason and the page the reason came from. Scores: rules 3, examples 4, inputs 5 (public websites), damage if wrong 5 (internal only). It ranks first.
Step 2, answer key: the founder had already tiered 50 firms. Reasons that repeated: "runs payroll for small business clients" (Priority 1), "general accounting firm that lists payroll as one of many services" (Priority 2), "software vendor, recruiter or not an accounting firm" (Not a fit), "branch of a large national network, so software is chosen at head office" (down one tier). Twenty rows went to the build tab, thirty to the test tab, including six where the website was down or unclear (should hand over).
Step 3, spec rules (excerpt): 1. Priority 1 if the firm's own site offers payroll services to small business clients. 2. Priority 2 if payroll is listed as one of several services without detail. 3. Not a fit if it is not an accounting or bookkeeping firm. 4. Move down one tier if the site says it is a branch of a large national network. 5. Hand over if the site cannot be read or does not describe any services.
Step 4, flow: mostly a straight line (row added, read homepage, decide, write, log) with one small loop: if the homepage does not list services, read up to two more pages (services, about) before deciding.
Step 5, tools: read_web_page (max three pages), update_record (writes tier, reason, evidence URL only, approval required during pilot), hand_over.
A test case, input and output:
INPUT case_id T-07 | firm: "Westfield & Co Accountants" | url: westfield-accountants.example
PAGES READ: homepage, /services
OUTPUT {"decision": "Priority 2",
"reason": "General accounting firm; payroll is one of eight listed services.",
"evidence": "/services lists tax, audit, bookkeeping, payroll and four others",
"confidence": "high"}
ANSWER KEY: Priority 2 -> pass
Step 8, first test run (illustrative): 24 of 30 matched. Reading the six failures showed one rule gap (payroll software resellers that describe themselves as "payroll specialists" were tiered Priority 1 instead of Not a fit), two missing inputs (the services page was never read because the homepage looked clear) and three correct hand-overs that the answer key had wrongly marked as Not a fit. The team added a rule ("if the site sells software rather than accounting services, Not a fit"), told the agent to check the services page when the homepage mentions more than one activity, corrected the three answer key rows and reran: all six should-hand-over cases were handed over and agreement rose to a level the founder accepted.
Step 9, pilot: two weeks on new list rows, founder still tiering by hand each morning and filling the human_verdict column. Most disagreements were on firms with very thin websites, so "fewer than three sentences about services" became a hand-over trigger.
Step 10, live: the agent now writes tiers directly for Priority 1 and Not a fit, and Priority 2 rows go to the founder's review queue. Priority 1 firms get a multi-channel sequence; Priority 2 firms get a lighter one. The founder reviews the log on Monday mornings.
Compliance and data: check local rules
- If the agent processes personal data (names, emails, phone numbers, call recordings), the data protection law where you and the people are applies. In the UAE, the federal Personal Data Protection Law (Federal Decree Law No. 45 of 2021) sets controls on processing personal data and companies' obligations to keep it secure and confidential, according to the UAE government portal. The same portal also lists a separate Data Protection Law for the Dubai International Financial Centre (DIFC Law No. 5 of 2020). Check which rules apply to you, and ask a lawyer if the agent handles sensitive data.
- Read the data terms of your model provider and platform: where data is processed, whether it is used for training and how long it is kept.
- Give the agent the minimum data it needs. A tiering agent needs firm names and websites, not personal phone numbers.
- If the agent talks to people (email, chat, phone), say that it is an AI. We do this on every agent we build, and some places require it.
- Keep the run log. It is how you answer "why did the system do that?" for a customer, a colleague or a regulator.
Cost and trade-offs
Running costs are usage based: model tokens, platform operations or executions, and any paid data or tools the agent calls. Building costs are your team's time or a provider's fee. We do not quote figures because they change with every platform's pricing; instead, measure cost per case in the pilot (Step 9) and compare it with what the task costs today in people's time. The main trade-offs: a larger model is usually more accurate and more expensive per case, so test a smaller one on your 30 cases before choosing; more tools make the agent more capable and harder to test; and full autonomy saves review time but raises the cost of each mistake, which is why go-live is staged.
What to measure and how to read it
| Metric | How to calculate | How to read it | What to do if it looks wrong |
|---|---|---|---|
| Live agreement | Runs with human_verdict "agree", divided by runs reviewed | Should stay close to your test agreement | Add the disagreements to the test tab, fix, rerun all tests |
| Hand-over rate | Runs handed over, divided by all runs | A stable rate means the rules fit your real cases | Rising: new case types or a broken input. Near zero: check that it is not guessing |
| Correction rate after go-live | Automated actions a person later changed, divided by automated actions | Your real error rate on live work | Put that case type back behind approval until fixed |
| Cost per case | Platform plus model usage for the period, divided by cases handled | Compare with the people-time cost of the same task | Trim tool output, lower max steps, test a smaller model |
| Owner time saved | Minutes per item before, minus review minutes per item now, times weekly volume | The business case for the agent | If review takes as long as doing the task, narrow the agent's scope |
| Downstream result | The business number the task feeds (replies, bookings, tickets closed) | Shows whether better decisions changed the outcome | If agreement is high but results do not move, the task or the rules are wrong, not the agent |
Pre-launch checklist
- One task, one owner and a written finish line.
- Answer key with 20 build rows and 30 held-back test rows, including hand-over cases.
- Spec signed off by the owner, including forbidden actions and hand-over triggers.
- Permission table: every write tool behind approval for the pilot; nothing deletes, pays or sends without a person.
- Prompt saved with a version number; prompt_version recorded in every log row.
- Validation check rejects malformed output and routes it to hand_over.
- Max steps per run set.
- All 30 test cases run on the current version; every should-hand-over case handed over.
- Cost per case measured on 10 to 20 real cases.
- Separate API key or user for the agent; data terms of every tool checked.
- AI disclosure in place if it talks to people.
- Daily log review booked in the owner's calendar for the first two weeks.
Your first week, day by day
- Day 1: list candidate tasks and score them with Asset 2. Pick one and write its finish line.
- Day 2: collect 50 past examples into the answer key (Asset 3) and split them 20 and 30.
- Day 3: write the spec (Asset 4) and test it on a colleague with five build rows.
- Day 4: draw the flow, fill the permission table (Asset 5) and write the prompt (Asset 6).
- Day 5: build the flow on your platform piece by piece with the log (Asset 7).
- Day 6: run the 30 test cases, score them (Asset 8), fix one thing at a time and rerun.
- Day 7: start the shadow pilot on live cases and measure cost per case on the first 10 to 20.
Frequently asked questions
How do I build an AI agent for my business?
Pick one repeated task with a clear finish line, collect about 50 past examples with the right outcome, write a one-page spec, give the agent only the tools it needs, write one short system prompt, test it on 30 held-back cases, run it beside a person for a week or two, then go live in stages.
Can I build an AI agent without coding?
Yes, for most first agents. No-code automation platforms such as Make, or n8n if you are comfortable with a little more setup, let you connect a trigger, a model call, tools and a log without writing code. The hard part is the spec and the test set, not the code.
Which AI agent builder or platform should I use?
Choose after you have written the spec. If the task is a fixed sequence with AI steps, a no-code workflow platform is enough. If the agent must choose its own next steps or work across your files, look at an agent framework or a coding agent. Test any platform on your own 30 cases before committing.
How long does it take to build an AI agent?
For one narrow task, about a week of focused work to reach a tested agent, then one to two weeks of shadow pilot before it acts on its own. Collecting good examples usually takes longer than the build itself.
What is the difference between an AI agent and an automation workflow?
A workflow follows a path you fixed in advance, with AI used at certain steps. An agent decides its own next step, such as which page to read or which tool to call, until it reaches a goal or a step limit. Start with a workflow when the path is predictable.
How do I stop an AI agent from making costly mistakes?
Give it the fewest tools possible, keep every write action behind approval during the pilot, let it hand over when unsure, validate its output before any action, set a step limit, log every run and rerun your test cases after every change.
Sources and further reading
- Anthropic, "Building effective agents" (workflows versus agents, start with the simplest solution, test in sandboxed environments with guardrails). Accessed 20 November 2026.
- Anthropic, "Writing effective tools for agents" (few targeted tools, clear names and parameters, concise responses, helpful errors). Accessed 20 November 2026.
- Anthropic, "Demystifying evals for AI agents" (start with 20 to 50 real tasks, unambiguous test cases, balanced sets, grade outcomes, read transcripts). Accessed 20 November 2026.
- Anthropic, "Effective context engineering for AI agents" (giving agents only the context they need). Accessed 20 November 2026.
- UAE Government portal, "Data protection laws" (Federal Decree Law No. 45 of 2021 on the Protection of Personal Data). Accessed 20 November 2026.
- SFC AI Stack page (tool descriptions; some links are referral links).
- SFC internal agent build notes and enrichment SOP (answer key and validation method, pilot on a small batch and estimate cost before scaling; methods adapted and anonymised, no client data used).
What to do next
Do Day 1 today: list the tasks that eat your week and score them. The top row is your first agent. If you would rather have it designed, built in your own accounts, tested against your answer key and handed over, see how we build AI outbound agents and AI voice agents, or book a short call to talk through the task you want to hand over.
