Compare two harbor runs on one task

Use when given two harbor run names and a task name, and the goal is to understand

why the two runs differ on that task — not just that they differ.

Inputs

• RUN_A: harbor run name (e.g. sonnet46-full)
• RUN_B: harbor run name (e.g. pi-sonnet46-full)
• TASK: bare task name (e.g. extract-elf, not terminal-bench/extract-elf)
• RUNS_DIR: defaults to evals/harbor/runs/ relative to the repo root

Procedure

1. Find each run's trial directory for the task

Harbor 0.8 names trial dirs <task>__<random-suffix> (e.g.

extract-elf__bU3GHs4), not <task>.1. The suffix is unique per trial,

so don't guess it — discover it from disk:

TRIAL_A_DIR=$(ls -d "$RUNS_DIR/$RUN_A/${TASK}__"*/ 2>/dev/null | head -1)
TRIAL_B_DIR=$(ls -d "$RUNS_DIR/$RUN_B/${TASK}__"*/ 2>/dev/null | head -1)

If either is empty, that run didn't include this task — stop and say so.

(ls "$RUNS_DIR/$RUN_A/" shows what's there.)

If you want to confirm the match, every result.json carries task_name

and trial_name:

jq '{task_name, trial_name}' "$TRIAL_A_DIR/result.json"

2. Headline facts

The fastest path is to let cmd.py task do it for you — it already prints

status, reward, duration, tokens, turns, cost, error class, and the tail of

the verifier stdout:

./evals/harbor/cmd.py task "$RUN_A" "$TASK"
./evals/harbor/cmd.py task "$RUN_B" "$TASK"

Only drop to raw jq against result.json if you need a field cmd.py task

doesn't print. The actual shape (harbor 0.8 TrialResult):

jq '{
  reward: (.verifier_result.rewards.reward
           // (.verifier_result.rewards | to_entries | .[0].value)
           // null),
  rewards_all: .verifier_result.rewards,
  duration_seconds: ((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)),
  input_tokens: .agent_result.n_input_tokens,
  cache_tokens: .agent_result.n_cache_tokens,
  output_tokens: .agent_result.n_output_tokens,
  cost_usd: .agent_result.cost_usd,
  error_type: .exception_info.exception_type,
  error_message: (.exception_info.exception_message // "" | split("\n")[0])
}' "$TRIAL_A_DIR/result.json"

The reward fallback mirrors reporter.trial_reward: if the verifier

didn't use the conventional reward key, take the first value in the

rewards map.

Derive status from those:

• pass if reward >= 1.0
• partial if reward > 0 (and < 1)
• fail if reward == 0
• timeout if reward is 0/null and error_type contains "timeout"
• error if reward is 0/null and error_type is set (non-timeout)
• no-reward if neither verifier_result.rewards nor exception_info is set

Reward wins over errors: harbor can record an AgentTimeoutError after the

verifier already scored a pass (the agent finished the work then the harness

timed out during teardown, or it timed out after writing the correct answer).

If we got points, count them. See reporter.trial_status for the canonical

rule.

Several agent_result fields can be null depending on the harness

(notably n_cache_tokens, n_output_tokens, cost_usd on some goose

runs). Don't treat that as a failure — just omit those facts from the

comparison if missing on either side. cmd.py task already applies

harbor's fallbacks (reading goose's complete event from agent/goose.txt

when the structured field is null), so its numbers are the right ones to

report.

3. Read the task spec

The task definitions are NOT in the harbor Python package. They are plain

text files on disk, in harbor's task cache. Do not run find / or

pip show harbor — that is the wrong direction.

Harbor caches under ~/.cache/harbor/ on every platform (it uses

Path("~/.cache/harbor").expanduser() unconditionally — there is no

~/Library/Caches/harbor on macOS, despite what you might expect).

The on-disk layout for package-backed tasks (the common case — everything

in terminal-bench/terminal-bench-2 lands here) is:

~/.cache/harbor/tasks/packages/<org>/<task>/<digest>/

Note: no dataset name in the path. Tasks are keyed by org + task name +

content digest, not by which dataset pulled them. The <digest> segment

changes when the task is republished, so discover the dir rather than

hardcoding:

TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/"$TASK"/*/ 2>/dev/null | head -1)
echo "$TASK_DIR"
ls "$TASK_DIR"

If that's empty, the task could be from a different org or a git source —

broaden the search. find returns the parent (one level above the

digest), so descend one more level. Guard against $PARENT being empty,

otherwise the glob expands to /*/ and matches the filesystem root:

PARENT=$(find ~/.cache/harbor/tasks -type d -name "$TASK" 2>/dev/null | head -1)
if [ -n "$PARENT" ]; then
  TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1)
fi

If both lookups come up empty, the task hasn't been downloaded on this

machine — bail out and report that, rather than guessing. (Runs sync via

cmd.py pull but the task cache does not, so a machine that only inspects

results may never have the spec locally.)

~/.cache/harbor/datasets/ exists too but holds dataset-level metadata,

not the per-task instruction.md / tests/ / solution/ files — not

what you want here.

Inside, you care about three files:

• instruction.md — exactly what the agent was asked to do
• tests/test_outputs.py (or sometimes run-tests.sh) — what the verifier

actually checks, line by line

• solution/solution.sh — the reference correct answer

Without all three you can't tell whether a wrong answer was a misread, a

shallow bug, or a verifier surprise. Quote the assertion that failed

when you describe a failure — paraphrasing is how wrong conclusions sneak in.

4. Read each agent's trajectory

Two sources, prefer the first when present:

• $TRIAL_DIR/agent/trajectory.json — harbor's ATIF format, one entry per

agent step. jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}'

gives a compact view. Most current runs have it; some older harness

versions may not.

• $TRIAL_DIR/agent/<harness>.txt — raw stream-json or log. The filename

matches the harness (commonly goose.txt or pi.txt; other harnesses

use their own name). Don't guess — run ls "$TRIAL_DIR/agent/" and use

whatever .txt file is there.

Skim, don't quote in full. For each agent identify:

• the approach it took (e.g. "wrote a Python script that walks the ELF section

headers")

• the final artifacts it left in the container (file paths it created or

modified)

• for losers, the failure mode — one of:
• misread the spec (wrong assumption about input/output)
• right approach, shallow bug (off-by-one, wrong encoding, wrong base address)
• ran out of clock (timeout) — note whether it was still making progress or

had gone in circles

• diverged into an unproductive thread (e.g. debugging a non-issue)
• the verifier expected something the spec didn't telegraph

5. Read the verifier output

$TRIAL_DIR/verifier/ typically contains:

• test-stdout.txt — the verifier's full stdout (assertion failures, pytest

output, etc.). This is usually the most diagnostic file.

• reward.txt — the scalar reward as a string.
• ctrf.json — structured test results in CTRF format, useful if you want

per-assertion pass/fail without grepping stdout.

tail -50 "$TRIAL_DIR/verifier/test-stdout.txt"

This is often more diagnostic than the agent log — it tells you exactly which

assertion failed and what the agent's output was at that point.

6. Produce the comparison

Output markdown with these sections in order:

• Headline (1 line): who won, by how much (reward + cost / duration if

meaningful, omitting fields that are null on either side).

• What A did (2-4 sentences): plan, final artifact, verifier outcome.
• What B did (2-4 sentences): same shape as A.
• Why outcomes differ (2-4 sentences): the actual mechanism. Not "B was

smarter" but "B's script used nm -n so its addresses matched the verifier's

ground truth, A's script used PIE-relocated virtual addresses which the

verifier doesn't normalize".

• Generalizable lesson (optional, 1-2 sentences): is this a pattern that

probably affects other tasks, or a one-off accident of this verifier? Skip

if unclear from one task.

Tools you'll need

• ./evals/harbor/cmd.py task <run> <task> for the headline numbers
• ls -d to discover the <task>__<suffix> trial directories
• jq for any result.json field cmd.py task doesn't print
• file reads against $TRIAL_DIR/agent/ and $TRIAL_DIR/verifier/
• find ~/.cache/harbor/tasks to locate the task spec

No Python imports, no harbor package required. Everything you need is on

disk as JSON / text files.