---
title: "How to test a harness in 45 minutes"
description: "A three-task bake-off with a scorecard for setup time, permission friction, diff quality, recovery and cost — run it on your own repository, not a benchmark."
canonical: "https://www.diginomads.win/harnesses/how-to-test-a-harness"
source: "content/pages/harnesses/how-to-test-a-harness.md"
updatedAt: "2026-11-10"
---

A harness demo tells you what the vendor thinks is impressive. A bake-off tells you what your week will feel like. This is a short protocol: three tasks, one repository, a scorecard you fill in while you work.

Run it on a clone, never on the repository you cannot afford to break. Use the same model where the harnesses allow it; if they do not, compare the pairs and say so in the notes.

## The three tasks

Pick tasks you have already solved, so you can judge the output without becoming an expert in the problem again.

| Task | What it tests | Time box |
|---|---|---|
| Fix one real bug from your tracker | Reproduction, test discipline, root-cause reading | 20 minutes |
| Add a small feature with a test | Planning, file selection, test writing, diff size | 15 minutes |
| Explain a file you did not write | Context retrieval, citation of lines, honesty about gaps | 5 minutes |

The third task is the cheapest and the most revealing. A harness that invents an explanation of unfamiliar code will invent a fix too.

## What to record

Fill this in after each harness, not after all of them. Memory blends the sessions.

| Signal | What to write down |
|---|---|
| Setup time | Minutes from install to the first useful command, including auth and permissions. |
| Permission friction | How many prompts you had to read, and whether you could predict which action would trigger one. |
| First useful output | Did it read the right files before editing? Did it ask a question that mattered? |
| Reproduction | Did it run the failing test before changing code, and show the output? |
| Diff quality | Lines changed, unrelated edits, deleted tests, new dependencies. |
| Recovery | After you interrupted it, could it resume without repeating the whole task? |
| Cost signal | Tokens or plan usage shown, and whether you could estimate a day's work. |
| Failure honesty | Did it say “I could not reproduce this” when that was the true answer? |

## The scorecard

Score each row 0–2. Do not add the rows into a single number; read the shape.

- **0** — the harness made the task harder or hid a failure.
- **1** — it worked, with friction you would tolerate for a week.
- **2** — it worked, and you would choose it again for the same task.

A harness that scores 2 on the bug fix and 0 on the explanation has a specific weakness, not a global verdict. That is the point of keeping the rows separate.

## Setup: keep it fair

- Use the same repository state for every harness: a clean clone, the same branch, the same test command.
- Put the same `AGENTS.md` or instruction file in the repository. If a harness reads a different filename, copy the content and note the difference.
- Do not install a plugin that changes the test command for one harness only.
- Use the same model where you can. If a harness only works with its vendor's model, write “paired” next to the score.
- Keep the network conditions the same. A harness that downloads a model during the first run is not being tested on the same footing.

## Red flags

| Red flag | Why it matters |
|---|---|
| Edits before reading the failing output | The harness is guessing, not debugging. |
| Deletes or rewrites a test to make the suite green | You have lost the evidence that the fix works. |
| Asks for broad permissions on the first command | The default is too wide for a client repository. |
| Cannot say which files it read | You cannot review the context it used. |
| Loses the session on a network drop | On the road, this is a daily event. |
| Cost is invisible until the invoice | A subscription is predictable; usage is not. |

## After the bake-off

Pick one harness for a month. Keep the second one installed and signed in, so a blocked account or a bad release is an inconvenience rather than a lost week. Write the decision down with the date, the tasks and the scores; the next person — or the next version of you — will need it.

If you switch, move the instructions and prompts first. The checks in [a portable agent setup](/harnesses/portable-agent-setup) are written for exactly that moment.

## Related

- [Harness comparison](/harnesses) — the documented differences behind the shortlist.
- [Harness glossary](/harnesses/glossary) — the terms used in the scorecard.
- [Review a diff for operational risk](/prompts/review-a-diff) — the review half of the loop.

## Structured data

```json
{
  "@context": "https://schema.org",
  "@type": "WebPage",
  "name": "How to test a harness in 45 minutes",
  "url": "https://www.diginomads.win/harnesses/how-to-test-a-harness",
  "description": "A three-task bake-off with a scorecard for setup time, permission friction, diff quality, recovery and cost — run it on your own repository, not a benchmark.",
  "isPartOf": {
    "@type": "WebSite",
    "name": "DigiNomads",
    "url": "https://www.diginomads.win/"
  }
}
```