Tool 01 · Agent Necessity Test

Should this be an agent at all?

Ten questions about one workflow, and a verdict on a five-stop spectrum — from write this in code, not AI, to supervised multi-agent with a human in the loop. Three of the five verdicts tell you not to build an agent. The complete scoring table is printed further down so you can check we are not pushing you up it.

Why this test exists

Reliability does not add up. It multiplies down. At 95% per step, an agent chaining steps degrades fast — which is why step count, not ambition, decides the shape.

1 step: 95%
3 steps: 86%
5 steps: 77%
9 steps: 63%
15 steps: 46%

End-to-end success, 95% per step

01The test

Ten questions about one workflow.

Answer for a single workflow, not a department. Nothing is stored, nothing is sent, and there is no email gate — the verdict is complete on this page the moment you answer.

The spectrum

Autonomy demand, left to right

Three of these five stops say don’t build an agent. T0 is costed below our own $25,000 minimum engagement.

Verdict · T0Provisional — 0 of 10 answered.

Rules or a script

This is a job for code, not for a model.

Nothing in your answers asks for language understanding, open-ended planning or judgement under ambiguity. A model here adds latency, cost and a failure mode you did not previously have, in exchange for capability the workflow never uses. Write the rules down and run them.

What to build instead

A scripted job or scheduled task against the existing APIs
A validation layer with explicit rules and a clear error queue
Structured logging so the exceptions become visible and countable

This verdict says don't build an agent. That is the more valuable of the two answers we can give you — talking someone out of a build is how we get trusted with the next one. Bring it and we will argue the other side properly.

Get a second opinion before you spend

We reply within one business day.

Our rubric, not an industry benchmark. We have no peer cohort and do not claim one — every weight, threshold and override is printed in section 03 so you can argue with it.

02The reading

Reliability does not add up. It multiplies down.

Almost every buyer reasons that a model which is 95% right produces a system that is 95% right. It does not. The two meters below decide the tier and price it; the panel underneath is the arithmetic that decides whether the thing ever reaches production.

The two axes

Autonomy picks the tier · control prices it

Autonomy demand: 0 / 100 · 0 of 168 pts

Decides which of the five tiers you land on.

Control demand: 0 / 100 · 0 of 68 pts

Never lowers the tier. It adds mandatory components and lifts the build band.

Collapsing these two into one maturity score is the category’s core modelling error. Heavy regulation does not mean “don’t use an agent” — it means you need an evaluation harness, a human-in-the-loop gate and an audit trace, which is exactly where budgets die.

Where the score came from — every question’s contribution

Every question, the answer you gave, the points it contributed and the maximum it could have contributed.
Q	Question	Your answer	Max
01	Input structure	Unanswered — scores zero	30
02	Path variability	Unanswered — scores zero	36
03	Exception rate	Unanswered — scores zero	26
04	Judgement type	Unanswered — scores zero	26
05	System reach	Unanswered — scores zero	16
06	Steps per item	Unanswered — scores zero	12
07	Regulatory exposure	Unanswered — scores zero	40
08	Cost of one wrong output, unreviewed	Unanswered — scores zero	28
09	Latency budget	Unanswered — scores zero	4
10	Rule churn	Unanswered — scores zero	18

Reliability panel

P = r^S · S = 1 · r = 95.0%

End-to-end: 95.0%r^S = 95.0% ^ 1 unattended success

Rescue queue: 250items a month landing on a human · volume × (1 − P) at 5,000/mo

Required per step: 95.00%to reach a 95% end-to-end target · target^(1/S)

Move steps into code: 0.0 pts95.0% → 95.0% · rescue 250 → 250 a month. Same model, same prompt.

Arithmeticnot an assumption. The formula is printed beside every figure.

The last card is the whole thesis of the business expressed as arithmetic: production reliability is an architecture problem, not a model-selection problem. On tau-bench, state-of-the-art function-calling agents succeed on under 50% of tasks, and pass^8 falls below 25% in the retail domain (Yao et al., arXiv:2406.12045). Published context — never used as a coefficient here.

Assumptions you can change3 coefficients

Three coefficients, all yours. Everything else in this tool is either a published rubric weight (printed in section 03) or arithmetic with the formula on screen. There are no hidden numbers.

A2 · Steps you could move out of the model loop into deterministic code

of 1

Answer Q06 (steps per item) to enable this. With one step there is nothing to move.

Assumptionyours to set(your architecture decision, not a measurement)

1 steps still inside the model loop

A1 · Per-step reliability

%

An assumption, not a benchmark. If you have error logs from the humans doing this work today, use their number.

Assumptionyours to set(An assumption, not a benchmark. If you have error logs from the humans doing this work today, use their number.)

A3 · Escaped-error ceiling you would sign up toFeeds the evidence card only

The share of items you would accept reaching a customer wrong, unreviewed. It changes how much evidence you need, not the verdict.

Assumptionyours to set(the ceiling you are prepared to be held to)

Evidence card

N = ⌈ ln(1 − confidence) ÷ ln(1 − tolerance) ⌉

Clean test cases needed: 299to demonstrate a 1% escaped-error ceiling at 95% confidence with zero observed failures

Rule-of-three check: 3003 ÷ tolerance — the standard approximation (Hanley & Lippman-Hand, JAMA 1983)

Arithmeticnot an assumption. The formula is printed beside every figure.

Before you can claim a 1% escaped-error ceiling you need roughly 299 clean test cases with zero failures. Most pilots ship with twenty.

Architecture for this verdict

T0 · Rules or a script

Components in

Deterministic rules
Schema validation
Structured logging
An exception queue

Components out

Any model call
Retrieval
Planner
Evaluation harness
Agent orchestration of any kind

Naming what you do not need is the point. Each of these is a line item somebody would otherwise quote you for.

Build band

Build cost only — run cost belongs to the configurator

Build band · T0: Below $25,000This models below our published $25,000 minimum engagement — smaller than we take on. This is your own team's fortnight, or a short advisory.
Assumptionyours to set(A modelling assumption. At full control demand it lifts the top of the band by 35%, covering the evaluation harness, the human-in-the-loop gate and the audit trace that regulated work requires.)

Over-engineering penalty: $25,000The tier above yours (T1 — Deterministic workflow / RPA) starts at $25,000 against $0. That is build you would not get back, before a single token.

Gartner forecasts that over 40% of agentic AI projects will be cancelled by the end of 2027 on escalating cost, unclear business value or inadequate risk controls (poll of 3,412).

PublishedGartner press release, June 2025, 26 Jul 2026

This tool deliberately does not model annual run cost. Run cost scales with volume × steps and is owned by the architecture configurator. Modelling it in two places guarantees the two tools drift.

03The rubric

Every weight, printed.

Our rubric, not an industry benchmark. We have no peer cohort and we do not claim one — so transparency substitutes for scale. Nothing below is hidden inside the tool, and the tool scores from exactly these numbers.

Self-interest disclosure

The tiers at the top of this scale are our most expensive engagements. The full scoring table is below so you can check we are not pushing you up it.

Question weights

Autonomy -20 to 168 pts · control 0 to 68 pts

Two orthogonal axes, never one maturity score. The autonomy total decides which tier you land on; the control total never lowers the tier — it adds mandatory components and lifts the build band. An unanswered question scores zero, which biases the verdict down.

The complete Agent Necessity Test scoring rubric: every question, every option and every weight.
Q	Question	Axis	Option	Weight
01	Input structure	Autonomy	(a)Structured records — database fields, API payloads, form submissions	0
			(b)Semi-structured documents with a known layout — invoices, claim forms	+8
			(c)Mixed formats, some scanned or photographed	+16
			(d)Free-form language — email, tickets, clinical notes, call transcripts	+24
			(e)Multi-modal — text plus image, audio or documents together	+30
02	Path variabilitythe heaviest question in the rubric	Autonomy	(a)One path, always identical	0
			(b)A handful of known branches, under ten, all documented	+6
			(c)Dozens of branches — enumerable in principle, nobody has	+16
			(d)The path depends on what you find as you go	+28
			(e)Nobody in the business can enumerate the steps in advance	+36
03	Exception rate	Autonomy	(a)Under 2	0
			(b)2 to 10	+5
			(c)10 to 25	+12
			(d)25 to 50	+20
			(e)Over 50	+26
04	Judgement type	Autonomy	(a)Following a documented rule	0
			(b)Applying a rule nobody ever wrote down	+10
			(c)Weighing several sources and trade-offs	+18
			(d)Making a contested call two of your own experts could disagree on	+26
05	System reach	Autonomy	(a)One, read-only	0
			(b)One, read and write	+4
			(c)Two to four systems	+10
			(d)Five or more, or a system with no API	+16
06	Steps per item	Autonomy	(a)1 to 2sets S = 2	0
			(b)3 to 5sets S = 4	+2
			(c)6 to 12sets S = 9	+5
			(d)13 to 25sets S = 18	+9
			(e)25 or moresets S = 30	+12
07	Regulatory exposure	Control	(a)None — internal convenience	0
			(b)Internal policy or audit only	+12
			(c)Contractual or financial obligation — SOX, a customer SLA	+22
			(d)Sector-regulated — HIPAA, GxP, FDA/MDR, financial conduct	+32
			(e)Safety-critical — an error injures someone or stops a line	+40
08	Cost of one wrong output, unreviewed	Control	(a)Trivial — someone notices and redoes it	0
			(b)Rework — hours of someone's time	+8
			(c)Money moves — a payment, price or shipment goes wrong	+18
			(d)Legal, clinical or safety harm	+28
09	Latency budget	Autonomy	(a)Sub-second — inside a keystroke or a machine control loop	−20
			(b)A few seconds, interactive	−6
			(c)Minutes	0
			(d)Hours, or an overnight batch	+4
10	Rule churn	Autonomy	(a)Once a year or less	0
			(b)A few times a year	+5
			(c)Monthly	+11
			(d)Constantly — per customer, per jurisdiction	+18

Meters: autonomy A = round(max(raw, 0) ÷ 168 × 100), clamped 0–100 · control C = round(raw ÷ 68 × 100), clamped 0–100.

Verdict bands and build bands

Five tiers · three say don’t build

The autonomy meter alone decides the band. Build figures are build cost only — this tool does not model annual run cost, because run cost scales with volume × steps and is owned by the architecture configurator. The band top is lifted by the control uplift and then clamped to our published $150,000 ceiling.

The five verdict tiers, their autonomy bands and their published build bands.
Tier	Verdict	Autonomy band	Build band
T0	Rules or a scriptThis is a job for code, not for a model.	0–14	below $25,000
T1	Deterministic workflow / RPAAutomate the path, not the thinking.	15–29	$25,000–$45,000
T2	One LLM call inside a deterministic pipelineOne model call, tightly fenced, inside code you control.	30–47	$30,000–$60,000
T3	Retrieval-grounded single agentOne agent, grounded in your corpus, choosing its own tools.	48–69	$55,000–$105,000
T4	Supervised multi-agent with human-in-the-loopSeveral specialists under a supervisor, with a human on the gate.	70–100	$90,000–$150,000

T0 is deliberately costed below our own $25,000 minimum engagement. It is smaller than we take on, and saying so is the point.

Overrides

7 rules · applied after banding

Applied after banding, in this order. Three of them are hard caps: answer “agentic” to everything but need a sub-second budget and you are capped at T1 and told exactly why. No override ever raises a verdict.

01Latency cap
caps at T1
Fired by Q09 — sub-second budget
A sub-second budget takes an LLM off the critical path. Anything model-driven runs alongside the loop, never inside it.
02Structured and stable
caps at T1
Fired by Q01 structured input, Q02 a fixed or documented path, Q10 stable rules
Structured input on a fixed path with stable rules is the textbook definition of a job for code — regardless of every other answer.
03Interactive cap
caps at T3
Fired by Q09 — a few seconds, interactive
An interactive budget rules out a supervisor's extra serial hops. One agent, or a router that dispatches once.
04Build nothing yet
blocking notice · tier unchanged
Fired by Q04 — a contested call two of your own experts could disagree on
Two of your own experts would disagree on the answer. That means you have no ground truth, which means you cannot build an evaluation set, which means you cannot tell whether the system is working. Fix the ground truth first — that is a workshop, not a build.
05Decision support, not automation
components and pricing · tier unchanged
Fired by Q03 — over half the items need a human to think
When most items need judgement, the system's job is to prepare the decision, not to make it. The verdict stands, but the shape of it changes.
06Recommend only
components and pricing · tier unchanged
Fired by Q07 — safety-critical
The system may recommend. It never actuates. Human-in-the-loop is not a design option here, it is the design.
07Regulation prices it, it does not forbid it
components and pricing · tier unchanged
Fired by Q07 — sector-regulated or safety-critical
A deterministic post-check and a reproducible audit trace become non-optional line items. Regulation does not forbid an agent. It prices one.

Coefficients and formulae

Three are yours · the rest is arithmetic

Every number that materially moves the answer, with its one permitted label. Three of them are yours to set on the tool itself; the rest are arithmetic with the formula printed beside the figure.

Adjustable coefficients

A1 · Per-step reliability, default 95.0%
Assumptionyours to set(An assumption, not a benchmark. If you have error logs from the humans doing this work today, use their number.)
A2 · Steps moved out of the model loop, default 0
Assumptionyours to set(your architecture decision, not a measurement)
A3 · Escaped-error ceiling, default 1%
Assumptionyours to set(the ceiling you are prepared to be held to)
Monthly volume, default 5,000 items/month
Assumptionyours to set(Contributes nothing to the verdict. Volume does not change what this should be — it only changes how much the answer costs you.)
Control uplift on the build band top, 35% at C = 100
Assumptionyours to set(A modelling assumption. At full control demand it lifts the top of the band by 35%, covering the evaluation harness, the human-in-the-loop gate and the audit trace that regulated work requires.)

Formulae — arithmetic, not assumptions

End-to-end reliability: P = r^S — per-step reliability compounded across S serial steps.
Rescue queue: volume × (1 − P) — items a month that fail end to end and land on a human.
Required per-step rate: r = target^(1/S), solved against a 95% end-to-end target.
Deterministic lift: P′ = r^(S − k) — the same model and the same prompt, with k steps moved into code.
Evidence sizing: N = ⌈ ln(1 − 95%) ÷ ln(1 − tolerance) ⌉, with the rule-of-three approximation 3 ÷ tolerance shown alongside.
Build band top: published top × (1 + C ÷ 100 × 0.35), clamped to $150,000.
Over-engineering penalty: band bottom of the tier above − band bottom of your tier. Build terms only, no invented run costs.

04The limits

What this tool cannot tell you

An instrument that will not state its own blind spots is selling something. These are ours.

It cannot see your data quality — the single most common reason a well-specified build fails.
It cannot see your integration surface, only how many systems you counted.
It cannot see your organisation's appetite for change, which is what actually kills builds.

MIT NANDA (2025) found roughly 95% of enterprise GenAI pilots showed no measurable P&L return, with barriers that were organisational rather than model-related.

Not sure which instrument you need? Start at the toolkit, or read how all of this is modelled.

05Sources

Everything cited, with a live link.

Only real, sourced numbers appear anywhere in this tool. Where a figure is context rather than a coefficient, it says so in the same sentence.

Anthropic, Building Effective Agents
Workflows are predefined code paths; agents direct their own process. Avoid agents when latency or cost matter, or when the path is predictable.
OpenAI, A Practical Guide to Building AI Agents
Three positive criteria for an agent; otherwise a deterministic solution will do.
Gartner press release, June 2025
Over 40% of agentic AI projects forecast cancelled by end-2027 on escalating cost, unclear business value or inadequate risk controls. Poll of 3,412.
Yao et al., tau-bench (arXiv:2406.12045)
State-of-the-art function-calling agents succeed on under 50% of tasks; pass^8 below 25% in retail.
Ardent Partners, ePayables 2024
9% best-in-class versus 22% typical invoice exception rate. Context only.
Hanley & Lippman-Hand, JAMA 1983
The rule of three for zero-failure confidence bounds.
MIT NANDA, 2025
Roughly 95% of enterprise GenAI pilots show no measurable P&L return, with barriers that are organisational rather than model-related.
McKinsey Global Institute, 2023
60–70% of work hours technically automatable with generative AI. Used strictly as a ceiling, never as our coefficient.

Should this be an agent at all?

Ten questions about one workflow.

Step zero

Rules or a script

Reliability does not add up. It multiplies down.

The two axes

Reliability panel

Evidence card

Architecture for this verdict

Build band

Every weight, printed.

Question weights

Verdict bands and build bands

Overrides

01Latency cap

02Structured and stable

03Interactive cap

04Build nothing yet

05Decision support, not automation

06Recommend only

07Regulation prices it, it does not forbid it

Coefficients and formulae

What this tool cannot tell you

Everything cited, with a live link.