| Pass@184.1Reported 84.1 · Unit not specified Published Unconfirmed | Codex v0.147.0- Settings
- Codex v0.147.0 standard app-server mode with adapted tool schemas. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- Codex v0.147.0 scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@185Reported 85.0 · Unit not specified Published Unconfirmed | OpenCode v1.18.15- Settings
- OpenCode v1.18.15 build agent with shell and file tools and native task delegation. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- OpenCode v1.18.15 scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@186.1Reported 86.1 · Unit not specified Published Unconfirmed | Pi v0.84.2- Settings
- Pi v0.84.2 RPC mode with file and shell tools plus search extension. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- Pi v0.84.2 scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@190.3Reported 90.3 · Unit not specified Published Unconfirmed | mini-SWE- Settings
- mini_swe_v2 port with a single bash tool and submission marker. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- mini-SWE
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- mini-SWE scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@185.8Reported 85.8 · Unit not specified Published Unconfirmed | DeepSeek Harness Standard- Settings
- DeepSeek Harness Standard with the full SDK profile and 26 initial tools. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- DeepSeek Harness Standard
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- DeepSeek Harness Standard scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@185.8Reported 85.8 · Unit not specified Published Unconfirmed | DeepSeek Harness PTC- Settings
- DeepSeek Harness PTC with run_code and 24 underlying tools. Max reasoning effort; Linux containers, temperature=1.0, top-p=0.95, 1M context and max_steps=500 as stated in the report note. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- DeepSeek Harness PTC
- Protocol
- Official DeepSeek-V4.1-Flash report Table 4; N=3 samples per task on Terminal-Bench v2.1.
- Conditions
- DeepSeek Harness PTC scaffold; Max reasoning effort. Table does not print a percent sign. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@187.3Reported 87.3 · Unit not specified Published Unconfirmed | Claude Code v2.1.105- Settings
- Claude Code v2.1.105 native tool interface. Max reasoning effort; same scaffold and benchmark settings as Table 4. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 5; Max reasoning effort; row protocol from Table 4.
- Conditions
- Claude Code v2.1.105; table does not print a percent sign; Max reasoning effort per the source table title. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@188.4Reported 88.4 · Unit not specified Published Unconfirmed | Claude Code v2.1.238- Settings
- Claude Code v2.1.238 native tool interface. Max reasoning effort; same scaffold and benchmark settings as Table 4. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 5; Max reasoning effort; row protocol from Table 4.
- Conditions
- Claude Code v2.1.238; table does not print a percent sign; Max reasoning effort per the source table title. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@188Reported 88.0 · Unit not specified Published Unconfirmed | Claude Code v2.1.251- Settings
- Claude Code v2.1.251 native tool interface. Max reasoning effort; same scaffold and benchmark settings as Table 4. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 5; Max reasoning effort; row protocol from Table 4.
- Conditions
- Claude Code v2.1.251; table does not print a percent sign; Max reasoning effort per the source table title. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@187.6Reported 87.6 · Unit not specified Published Unconfirmed | Claude Code v2.1.259- Settings
- Claude Code v2.1.259 native tool interface. Max reasoning effort; same scaffold and benchmark settings as Table 4. This is an ordinary single-agent public scaffold run, not the Section 5.3.5 multi-agent experiment.
- Unknowns
- Public reproducible ordinary single-agent scaffold at explicit Max reasoning effort. Presentation uses the existing max mode; scaffold and version remain conditions and do not create a separate effort tier.
- Tools
- Tools allowed
- Reasoning budget
- Max
- Framework
- not recorded
- Protocol
- Official DeepSeek-V4.1-Flash report Table 5; Max reasoning effort; row protocol from Table 4.
- Conditions
- Claude Code v2.1.259; table does not print a percent sign; Max reasoning effort per the source table title. Ordinary single-agent public scaffold; not a cross-version average or multi-agent run.
| View this score in rankings |
|---|
| Pass@187.8Reported 87.8 · Unit not specified Published Unconfirmed | Claude Code four-version average- Settings
- Average computed from the unrounded Pass@1 scores of four Claude Code versions. Max reasoning effort; same scaffold and benchmark settings as Table 4.
- Unknowns
- Claude Code version-specific results are retained raw and are not mapped as ordinary effort tiers.
- Tools
- Tool access not recorded
- Reasoning budget
- Max
- Framework
- Claude Code harness
- Protocol
- Official DeepSeek-V4.1-Flash report Table 5; Max reasoning effort; row protocol from Table 4.
- Conditions
- Claude Code four-version average; table does not print a percent sign.
| |
|---|