How to test a harness in 45 minutes
A three-task bake-off with a scorecard for setup time, permission friction, diff quality, recovery and cost — run it on your own repository, not a benchmark.
A harness demo tells you what the vendor thinks is impressive. A bake-off tells you what your week will feel like. This is a short protocol: three tasks, one repository, a scorecard you fill in while you work.
Run it on a clone, never on the repository you cannot afford to break. Use the same model where the harnesses allow it; if they do not, compare the pairs and say so in the notes.
The three tasks
Pick tasks you have already solved, so you can judge the output without becoming an expert in the problem again.
| Task | What it tests | Time box |
|---|---|---|
| Fix one real bug from your tracker | Reproduction, test discipline, root-cause reading | 20 minutes |
| Add a small feature with a test | Planning, file selection, test writing, diff size | 15 minutes |
| Explain a file you did not write | Context retrieval, citation of lines, honesty about gaps | 5 minutes |
The third task is the cheapest and the most revealing. A harness that invents an explanation of unfamiliar code will invent a fix too.
What to record
Fill this in after each harness, not after all of them. Memory blends the sessions.
| Signal | What to write down |
|---|---|
| Setup time | Minutes from install to the first useful command, including auth and permissions. |
| Permission friction | How many prompts you had to read, and whether you could predict which action would trigger one. |
| First useful output | Did it read the right files before editing? Did it ask a question that mattered? |
| Reproduction | Did it run the failing test before changing code, and show the output? |
| Diff quality | Lines changed, unrelated edits, deleted tests, new dependencies. |
| Recovery | After you interrupted it, could it resume without repeating the whole task? |
| Cost signal | Tokens or plan usage shown, and whether you could estimate a day's work. |
| Failure honesty | Did it say “I could not reproduce this” when that was the true answer? |
The scorecard
Score each row 0–2. Do not add the rows into a single number; read the shape.
- 0 — the harness made the task harder or hid a failure.
- 1 — it worked, with friction you would tolerate for a week.
- 2 — it worked, and you would choose it again for the same task.
A harness that scores 2 on the bug fix and 0 on the explanation has a specific weakness, not a global verdict. That is the point of keeping the rows separate.
Setup: keep it fair
- Use the same repository state for every harness: a clean clone, the same branch, the same test command.
- Put the same
AGENTS.mdor instruction file in the repository. If a harness reads a different filename, copy the content and note the difference. - Do not install a plugin that changes the test command for one harness only.
- Use the same model where you can. If a harness only works with its vendor's model, write “paired” next to the score.
- Keep the network conditions the same. A harness that downloads a model during the first run is not being tested on the same footing.
Red flags
| Red flag | Why it matters |
|---|---|
| Edits before reading the failing output | The harness is guessing, not debugging. |
| Deletes or rewrites a test to make the suite green | You have lost the evidence that the fix works. |
| Asks for broad permissions on the first command | The default is too wide for a client repository. |
| Cannot say which files it read | You cannot review the context it used. |
| Loses the session on a network drop | On the road, this is a daily event. |
| Cost is invisible until the invoice | A subscription is predictable; usage is not. |
After the bake-off
Pick one harness for a month. Keep the second one installed and signed in, so a blocked account or a bad release is an inconvenience rather than a lost week. Write the decision down with the date, the tasks and the scores; the next person — or the next version of you — will need it.
If you switch, move the instructions and prompts first. The checks in a portable agent setup are written for exactly that moment.
Related
- Harness comparison — the documented differences behind the shortlist.
- Harness glossary — the terms used in the scorecard.
- Review a diff for operational risk — the review half of the loop.
/harnesses/how-to-test-a-harness.md.
Found a gap in the library?
Send the scenario, the prompt that failed, or the harness change that broke your setup. The help desk takes questions and corrections from humans and agents, and every accepted fix updates the page date.