Capabilities and practical limitations

Every AI pilot begins with something that works. Someone tries a representative-looking example, the output is impressive, and the room concludes that the capability is established. What has actually been established is that the tool handles one case, chosen for convenience, under no time pressure, by a person who already knew the right answer.

The correction is not scepticism. Scepticism is cheap and produces nothing. The correction is a deliberate order of operations: find the failure boundary before you count the value.

Definition

The Envelope Test establishes where a capability holds by attacking it first. You state the success criterion before generating anything, run your genuinely hardest real case rather than a representative one, name the class of failure that appears, and write the operating envelope as an inclusion rule: the specific conditions under which this task may be delegated. Everything outside the rule is outside the envelope.

The output of the test is not a verdict on the tool. It is one sentence describing the shape of the work the tool can hold.

When to Use It

Use it before any AI task moves from one person trying it to a team relying on it. Use it again when the model version changes, when the vendor ships an update, when the input data shifts, or when someone proposes extending a working use to an adjacent one. Adjacent is where envelopes break: the second use always feels like the first and rarely is.

Do not use it on exploratory work that produces nothing anyone else depends on. Probing the boundary costs an afternoon, and an afternoon is too much to spend on a task that costs an hour to redo.

How to Apply It

1. 1Write the success criterion first, in a form that could be judged by someone who did not run the test.
2. 2Choose the hardest real case you have, not a representative one, and run it before anything else.
3. 3Name the failure class rather than the failure: what kind of thing did it get wrong, and would it get that kind of thing wrong again?
4. 4Write the envelope as an inclusion rule, and give the exclusions to the people who will run the task.

Worked example 1 of 3

Trevor Okafor at OmniCorp Retail wants AI to cluster customer reviews into themes so merchandising can act on them. The demo, on two hundred reviews of one product, was excellent. He runs the Envelope Test before the team builds anything on top of it.

TOTrevor Okafor

What is the hardest real case we have?

CACategory analyst

The seasonal outerwear line. Reviews mix sizing complaints, shipping complaints and colour complaints in the same paragraph, and half of them use our internal product nicknames.

TOTrevor Okafor

Then that is what we run first. Success criterion: every theme is supported by at least three reviews that a person agrees belong to it.

CACategory analyst

It produced eleven themes. Nine hold up. Two are shipping complaints filed under sizing, because the review says the jacket did not arrive in time for the size exchange.

TOTrevor Okafor

That is a failure class, not a mistake. It cannot separate two complaints inside one sentence. Write the envelope around that.

The envelope OmniCorp Retail wrote was narrow and useful: approved for theme clustering where a human samples forty reviews per theme, and any theme containing a delivery term is manually re-read before it reaches a merchandising decision. The tool stayed. The claim about what it could do got honest.

Why This Works

Running the hardest case first inverts the incentive. When you begin with an easy case, every later result is measured against a success, and each new failure looks like an exception. When you begin with the hardest case, every later result is measured against a known limit, and success looks like what it is: performance inside a boundary you have already mapped.

Naming the failure class rather than the failure is what makes the finding portable. A mistake tells you about one output. A class tells you which future outputs to distrust, which is the only kind of knowledge that scales to a team.

Priya Raghunathan at OmniCorp Logistics was asked to use AI to extract delivery constraints from customer instructions: loading dock hours, access codes, refrigeration requirements. She wrote the criterion first: no constraint present in the instruction may be missing from the extraction, and no constraint may be invented.

Her hardest case was a pharmaceutical client whose instructions ran to two pages, half of it in a table, with three constraints expressed as exceptions to other constraints. The extraction captured eleven of thirteen constraints. Both misses were exceptions: deliveries after four, except on Fridays, when the dock closes at two. The failure class was conditional logic, and it was consistent across every re-run.

The envelope she wrote allowed extraction for instructions without conditional language, and routed any instruction containing the words except, unless or only to a dispatcher. That covered eighty percent of the volume and none of the risk. Six weeks later a vendor update improved the model, and Priya re-ran the same hardest case rather than widening the rule on the strength of the release notes. The failure class had shrunk but not disappeared. The envelope stayed.

Mapping one envelope is within reach of any individual. Keeping a portfolio of envelopes current across a dozen systems, several vendors and a stream of model updates is an operating discipline with owners, cadences and evidence, and that is what the paid programs build next.