Weight-space programming, measured
Who is the best weight-space programmer?
We plant known defects in a private, function-preserving Qwen3.5-0.8B reference. The Model track fixes the harness; the Agent track lets teams bring their own. Exact ground truth, public traces and public costs.
Leaderboard
Overall score is the mean across instances. Confidence intervals bootstrap instances.
Results pending
Leaderboard launching Nov 6
No official scores are published yet. Launch rows will include bootstrap confidence intervals, costs, token counts, GPU calls and public traces.
Launch roster
Model track
- DeepSeek V4.1 Flash?
- Qwen3.8 Flash?
- GLM-5.3 Flash?
- GPT-6 Luna?
- Grok 4.1 Fast?
- MiniMax M2.7?
- Gemini 3.5 Flash-Lite?
- GPT-6.1 Terra?
Agent track
- Cursor Agent · Composer 2.5?
- Cursor Agent · Grok 4.7?
Baselines
- Oracle100
- Red-team harness?
- Scripted pipeline?
- No-op0
Score vs. cost
Higher and farther left is better. The dashed line marks the model Pareto frontier; cost uses a logarithmic axis.
What this does not show
A benchmark result is evidence about performance under these rules—not a broader metaphysical claim.
The planted failures are controlled tasks with exact ground truth, not a claim about every real model failure.
A high score means the model runs a find-direction → edit → verify loop effectively with the supplied tools.
Scores share a fixed budget, but provider pricing, token accounting, and default reasoning modes still matter.