held-out games (composite rules) · evals of frontier & open-source models (our trained checkpoints are in RL & SFT experiments below)
training games (primitive rules only) · oracle-guided trace collections
| run | RHAE mean±std | games | lvls | avg_lvls | |
|---|---|---|---|---|---|
| sft_fable_20260701 | 41.3 ± 27.0 (k2) | 91/155 | 608/775 | 3.92/5 | compare |
| sft_opusv5_20260720 | 36.4 ± 23.6 (k2) | 83/137 | 556/685 | 4.06/5 | compare |
our trained checkpoints (SFT/RL) evaluated with SAME harness — numbers on the held-out games are directly comparable with frontier/open-source rows above
| run | RHAE mean±std | games | lvls | avg_lvls | |
|---|---|---|---|---|---|
| v6.2_qwen359b-sft-v6eff_20260722 | 6.8 ± 2.5 (k2) | 3/20 | 45/100 | 2.25/5 | wandb · compare |
| v6.2_qwen359b-sft-v8full_20260722 | 8.0 ± 1.6 (k2) | 1/20 | 43/100 | 2.15/5 | wandb · compare |
| v6.2_qwen359b-sft-v8resetaware_20260722 | 7.7 ± 2.1 (k2) | 5/20 | 45/100 | 2.25/5 | wandb · compare |
| v7.3_qwen359b-base-rl-s5_20260723 | 8.9 ± 6.5 (k2) | 5/20 | 38/100 | 1.90/5 | wandb · compare |
| v7.3_qwen359b-base_20260723 | 4.1 ± 3.8 (k2) | 4/20 | 32/100 | 1.60/5 | N/A · compare |
| v7.3_qwen359b-sft-v8resetaware-rl-s15_20260723 | 13.4 ± 2.4 (k2) | 3/20 | 47/100 | 2.35/5 | wandb · compare |
| v7.3_qwen359b-sft-v8resetaware_20260723 | 10.6 ± 2.7 (k2) | 1/20 | 39/100 | 1.95/5 | wandb · compare |
play · record · rank — you are scored with the SAME official RHAE (k=2) as the models above, and every run you finish sharpens the empirical human-baseline steps behind the models' 'human' metric
| run | RHAE mean±std | games | lvls | avg_lvls | |
|---|---|---|---|---|---|
| human_Guanghan_20260724_162108 | 21.4 ± — (k2) | 1/4 | 8/20 | 2.00/5 | compare |
RHAE (Relative Human Action Efficiency): per level,
scoreL = min(115, (baselineL / actionsL)² × 100) if the level was completed, else 0;
a game's score is the completion-capped, level-weighted mean of scoreL (official bench.scoring, cap = 5 levels).
The cell shows the per-rollout mean over the bundle's games, mean ± std over seeds/rerolls, at the reporting convention k=2 (baselineL = 2 × BFS-optimal steps) — directly comparable to the published frontier tables.
The human metric re-scores the same runs with baselineL = average human steps from the HiScore runs (per level; levels no human has completed keep the k2 baseline).