ProgramBenchBrowse 284 families

ProgramBench

Can the model complete substantial programming work under this benchmark's agent setup?

Version / measure

Domain: Coding
Ranked models: 12
Reported values: 17
Reports: 4 · lab
Direction: Higher is better
Status: Version not specified

Rankings

Best reported score per model across effort settings

12 models

#	Model	Best setting	Score
1	Kimi K3Moonshot AIOW	maxcost n/a1 value · Jul 17 · source	77.8%
2	GPT-5.6 SolOpenAI	maxcost n/a1 value · Jul 17 · source	77.6%
3	Claude Fable 5Anthropic	maxcost n/a1 value · Jul 17 · source	76.8%
4	Claude Opus 4.8Anthropic	maxcost n/a3 values · Jul 17 · source	71.9%
5	GPT-5.5OpenAI	xhighcost n/a3 values · Jul 17 · source	70.8%
6	GLM-5.2Z.aiOW	maxcost n/a2 values · Jul 17 · source	63.7%
7	Kimi K2.7 CodeMoonshot AIOW	thinkingcost n/a1 value · Jul 22 · source	53.6%
8	GLM-5.1Z.aiOW	reportedcost n/a1 value · Jun 16 · source	50.9%
9	Kimi K2.6Moonshot AIOW	thinkingcost n/a1 value · Jul 22 · source	48.3%
10	DeepSeek V4 ProDeepSeekOW	reportedcost n/a1 value · Jun 16 · source	47.8%

Effort curve

Score vs. reported cost. Lines connect complete sweeps only.

0 values

No cost-linked effort sweep for this version.

Definition and comparison boundarypublic methodology

Can the model complete substantial programming work under this benchmark's agent setup?

Software implementation, debugging, or repository work under the published evaluation protocol. Higher is better. The value is the percentage reported in this lab's table.

Harness, tools, prompts, attempt count, and benchmark version can materially change the result. Compare within one reporting context.

DeepSeek's official V4.1-Flash release table; exact source benchmark label ProgramBench retained. Kimi Code harness. Claude Code 2.1.156; max reasoning effort for GLM-5.2. Kimi Code CLI with thinking enabled, temperature 1.0, top-p 0.95, and 262,144-token context; GPT-5.5 used Codex xhigh and Claude Opus 4.8 used Claude Code xhigh.

Method / source ↗