← runs

Harness Changelog

The agent harness (perception, prompts, reset semantics, scoring) is versioned; the HARNESS chips on the runs page group results by these conventions. Numbers from different versions are not directly comparable — rank within a chip, not across.

v82026-07-23 · merged, trace-validated (2 loop iterations), pending A/B

LLM-first game entry · discovery-by-ORAI

No reconnaissance stage at all: the game's first move comes from the model's own reflection. Button meanings start “not yet discovered” (submit stays told — a designed exception) and the prompt only ever transcribes what the model itself tried, with provenance (“observed ×3 this level” / “learned on an earlier level — unverified here”). Also: plan cap raised 5→16 and disclosed in the prompt.

Why: The L0 probe prefix was the last harness-owned tax: ~30 automatic steps against baselines of 12–15 — even perfectly-played first levels scored ~9–21/100. And by design intent, discovering what buttons do IS part of the task; the harness was doing it for the model. After v8, the only entity that touches the environment is the model.

v7.32026-07-23 · on branch, pending A/B

No blind replay at level entry

Entering a new level no longer auto-replays the previous level's winning actions; the first move comes from the model's own reflection.

Why: A pre-GT “maybe the same solution works” heuristic, believed dormant, was firing at nearly every level transition in every version ever run — ~5–12 blind scored actions per run, paid at the quadratic step tax. Found by the new post-eval trace audit (and independently in trace-viewing), not by score tables — scores can't see it because every arm paid it.

v7.22026-07-22 · on branch, pending A/B

Ground-truth legend + convention stamp

Board legend built from the engine's own color constants instead of exploration statistics; plan parser prefers the last clean action list; every run now stamps its resolved convention flags into logs, traces, and result JSONs.

Why: The old legend was exploration-dependent noise — the same game classified colors differently between levels (and labeled the drawn path as the agent). The stamp makes “thought it was arm B, ran arm A” structurally impossible.

v7 / v7.12026-07-22 · ADOPTED 2026-07-23 (A/B: opus 37.7→57.8, qwen9b 2.5→6.9)

LLM-first level entry · planless random fallback

v7: after the first level, no automatic re-probing — the model acts from reflection immediately, oriented by a level-transition breadcrumb and a Level K/M header. v7.1: a reflection that produces no plan takes ONE random action, then reflects again.

Why: The automatic level-entry steps were 18% of ALL actions for strong models (score is quadratic in actions — counterfactual +25 RHAE). Rules provably carry across levels, so re-probing bought orientation only. v7.1: env steps are the scored currency; LLM calls are free — stop paying steps to fill planless gaps.

v6.22026-07-22

Competition reset parity

Resets that would trigger the engine's full-game wipe (consecutive resets, reset-as-first-move) are not executed — the step still counts, nothing is erased.

Why: Mirrors the official competition grader, which skips exactly that wipe. Local evals were harsher than the competition: 103 full wipes across 300 frontier rollouts.

v6 / v6.12026-07-21 → 22

Ground-truth transition observations

Per-action observation diffs derive from the env's ground-truth path delta instead of pixel diffing (v6.1 removed a v6 side-effect that spammed redundant trail mentions).

Why: Perception is GIVEN in this benchmark, not inferred. Pixel diffs missed or mislabeled changes — models repeated actions because the harness told them nothing had happened.

v52026-07-16 · adopted via pre-registered A/B

Reflect on plan exhaustion

When the current plan runs out, the agent reflects immediately instead of taking filler steps until the reflection cadence comes due.

Why: Planless gaps burned scored steps on cadence filler. Opus +2σ in the A/B; also made v4's agent-chosen reset usable (v4 alone measurably hurt).

v42026-07-15

Agent-chosen RESET, official accounting

Level restarts became the agent's strategic choice (ACTION0 in the plan space, semantics stated in the prompt). All steps keep counting across restarts — no free eraser. The old harness-side retry machinery was deleted (provably inert: 0 retries granted in 100 rollouts).

Why: Restart-or-persist is part of the skill being measured, and the accounting now matches the official grader's.

v32026-07-02

Unified scoring conventions

One scoring convention everywhere: 5-level score cap, 250-call reasoning budget for every model, give-up ends the run (previously mislabeled as a level skip and rescored).

Why: Different caps and budgets had silently made cross-model tables incomparable — earlier numbers mixed /10 and /5 denominators and uncapped API budgets.

Full details live in the dated reports under research-artifacts/arc-witness-runs/reports/.