A few days ago, we argued that the real product in an AI feature isn't the model - it's the harness: the systematic layer that evaluates, constrains, and instruments what the model does before it's allowed to touch anything that matters. We closed with "the model is a component, the harness is the product."

A few of you DM'd us and asked the fair follow-up question: alright, but what does that actually look like when you build one? So based on that last piece, we thought we'd answer from the product side - a little of what we've actually done.

Because here's the honest version: six months ago, we didn't have the word "harness" either. We had pieces, under older and far less glamorous names: model routing, an evaluation corpus built from staff feedback, typed tools with real permissions, consent enforced as a constraint, human approvals on the actions that carry consequences. What we didn't have - and what we'd later name as the actual bar - was one system that tied all of it together. Getting to that one system is the part worth talking about.

The two harnesses were always the same harness

The sharpest point we made last time is that "harness" has been quietly used to mean two different things. There's the evaluation harness - the offline test bench that scores a model on defined tasks before you ship it. And there's the agent harness - the runtime scaffolding that constrains what the model can actually do once it's live. For a while these lived in different worlds, owned by different people, argued about in different meetings.

They are the same thing. An evaluation harness sits between the model and the decision to ship it. An agent harness sits between the model and the action it's about to take. Different placement, identical philosophy: you never let the raw model touch a consequence without a layer in between that defines the task, judges the output, limits the blast radius, and records what happened.

That used to be our gap, too. We kept evaluations in one place and runtime guardrails in another, and we've put words to exactly why that's slow - two disconnected stacks that happen to share a model, and a team relitigating quality by anecdote in every standup. So we closed the loop. The logic we use to score the agent offline is the same logic that can refuse or escalate a live turn. Offline and online became two faces of one instrument.

What you rent versus what you own

Here's the reframe that we think is the whole ballgame: the frontier model is rented.

The best models improve on someone else's schedule. Switching between them should be close to a configuration change after a regression run - and if it isn't, that's a design smell, not a fact of life. What persists across every model upgrade is the layer that encodes our judgment about what "good" and "safe" mean in our specific domain. For us that domain is consented sales conversations, where the difference between a helpful answer and a liability is often a single sentence.

The model is the part we rent. The harness is the part we own. And you can only own it if you build the part that's actually yours.

The four capabilities, in a domain that punishes vagueness

We named four things every real harness needs. What we didn't say last time: you don't really understand them until a domain forces you to.

Task definition. A task is not "help the customer." It's an input distribution, a success criterion, and an explicit scope boundary. Our agent can answer a visitor from real, grounded inventory. It cannot invent a discount, quote a price that isn't actually listed, claim a message was sent when no send happened, or commit the business to anything. Those are different jobs - some of them belong to a different task, and some of them require a human. If you can't write the task down, the model can't reliably hit it, and you can't tell when it misses.

Scoring. Staff feedback is human signal, not a ship gate. The gate is systematic: the same programmatic checks run on a historical example and on a live turn, and they fail the same way in both places. An unauthorized discount fails the same way. A dollar figure that doesn't trace back to real inventory fails the same way. That symmetry is the point - the score isn't a vibe, it's a rule that behaves identically whether you're testing or live.

Containment. The least glamorous capability, and the one that actually keeps you out of the newspaper. Typed tools with permissions enforced at the boundary, not politely requested inside a prompt. Consent respected as a hard constraint the model cannot argue its way around. Human approval required for the actions with real-world consequences. A model that can read is a feature; a model that can read and write and send and commit, with no ceiling, is an incident with a countdown.

Observability. Every scored turn writes a transcript: what the visitor said, what the model produced, which tools ran, what we scored, and what we allowed through. The failures write a record too, on purpose - so an incident is something you can reconstruct instead of reenact. A harness without observability is a black box wearing a lab coat.

"But our provider handles safety"

This is the objection we hear most, usually from the smartest person in the room. And it's a category error.

Provider guardrails are real and we want them. The leading models won't help build a weapon, won't spew slurs, won't walk someone through self-harm. That's the floor.

But a provider cannot know your domain, because it was never told about it. A sales harness has to stop the agent from promising a discount it can't authorize, quoting a price that isn't on the page, claiming a send that never ran, or making a commitment on the company's behalf. It has to respect what a visitor has and hasn't agreed to share. It has to hold a message until quiet hours end. None of that lives in a generic safety filter, and none of it ever will - because it's specific to us, our customers, and the rules our industry runs on.

The provider protects the world from the model. Our harness protects the customer's business from the model. Those are different jobs, and only one of them is anyone else's responsibility.

Build, buy, or hybrid - and where we landed

We said most serious teams end up hybrid, leaning build. We'd know - we lived it. You buy the commodity layers - a model to rent, the plumbing you'd be foolish to reinvent. You build the part that encodes your judgment: the task catalog, the shared scorer, the isolated evaluation runner, the inspectable transcript, and the live gate that decides whether a given turn ever reaches a customer.

We didn't write our own trace database. We did write our own definition of "did this agent do the right thing," because nobody outside our four walls can write that sentence for us.

The second question in every review

The first question is still "which model?" The honest answer is "whichever managed frontier model fits the feature and the complexity." The second question - the one we flagged last time, and the one we now hear constantly - is "what's the harness?"

That's the question where the answer is entirely ours. The model will be better in six months, and the switch should be nearly a config change. What we'll still own is the layer that decides whether that better model is allowed to speak for the business.

The model is a component. The harness is the product.

That was the thesis. This is the build.

#AI #ArtificialIntelligence #MachineLearning #AIEngineering #LLMOps #AIInfrastructure #AIAgents #LLM #AIEvaluation #AISafety #AIGovernance #FutureOfWork

Everyone Just Discovered the "AI Harness." We've Been Shipping One.