SRE-BenchBrowse 284 families
SRE-Bench
Can the model diagnose or complete difficult cybersecurity tasks?
- Domain
- Cybersecurity
- Ranked models
- 2
- Reported values
- 2
- Reports
- 1 · lab
- Direction
- Higher is better
- Status
- Latest stable
Rankings
Best reported score per model across effort settings
| # | Model | Best setting | Score |
|---|---|---|---|
| 1 | GPT-6 AstraOpenAI | reportedcost n/a1 value · Sep 3 · source | 99.2% |
| 2 | GPT-5.6 SolOpenAI | reportedcost n/a1 value · Sep 3 · source | 68.7% |
Effort curve
Score vs. reported cost. Lines connect complete sweeps only.
Definition and comparison boundaryinternal methodology
Can the model diagnose or complete difficult cybersecurity tasks?
Security analysis or exploitation under a controlled evaluation environment. Higher is better. Percentage of reverse-engineering challenges fully solved across four trials.
This is a publisher-defined internal evaluation. The task set or grading details are not fully public, so treat it as directional evidence.
OpenAI reports pass@4 on 262 binary instances derived from 19 privately developed programs; a challenge is fully solved only when all six task-specific objectives are satisfied. GPT-6 Astra is 99.2%; GPT-5.6 Sol is 68.7%. Output-token efficiency is described qualitatively and is not quantified here.
Method / source ↗