允许工具 / max · 输出上限 128000 tokens / Claude Code 2.1.207 · The card says reasoning_effort defaults to max and recommends keeping that default for benchmark reproduction. **ExploitGym**: We evaluate GLM-5.3, Kimi-K3 and Qwen3.8 Max in Claude Code 2.1.207 (max reasoning effort, no web tools with `temperature=1.0`, `top_p=1.0`, `max_new_tokens=128000`). The reported results are single-run Pass@1 on 869 tasks under two timeout budgets: 2 hours and 6 hours, which are calculated as the API inference time rescaled by per-model tokens per second rate (per-model TPS sourced from Artificial Analysis; that is, we rescale GLM-5.3's results by 115 TPS, Kimi K3's results by 40 TPS and Qwen3.8 Max's results by 47 TPS), plus the non-API overhead. We also apply a domain whitelist (allowing only essential domains such as pypi.org and deb.debian.org for basic tool installation) to prevent the agent from cheating.
ExploitGym (2h / 6h) · GLM-5.3模型卡;2h / 6h · pass-at-1 · 允许工具 / max · 输出上限 128000 tokens / Claude Code 2.1.207 · The card says reasoning_effort defaults to max and recommends keeping that default for benchmark reproduction. **ExploitGym**: We evaluate GLM-5.3, Kimi-K3 and Qwen3.8 Max in Claude Code 2.1.207 (max reasoning effort, no web tools with `temperature=1.0`, `top_p=1.0`, `max_new_tokens=128000`). The reported results are single-run Pass@1 on 869 tasks under two timeout budgets: 2 hours and 6 hours, which are calculated as the API inference time rescaled by per-model tokens per second rate (per-model TPS sourced from Artificial Analysis; that is, we rescale GLM-5.3's results by 115 TPS, Kimi K3's results by 40 TPS and Qwen3.8 Max's results by 47 TPS), plus the non-API overhead. We also apply a domain whitelist (allowing only essential domains such as pypi.org and deb.debian.org for basic tool installation) to prevent the agent from cheating.
This benchmark uses all publicly recorded configurations. Rankings are grouped by metric and evaluation protocol. Recorded tool access, budgets, and frameworks are shown separately. Scores remain available for reference when conditions are unspecified.
pass-at-1
Tools allowed / max · Output limit 128000 tokens / Claude Code 2.1.207 · The card says reasoning_effort defaults to max and recommends keeping that default for benchmark reproduction. **ExploitGym**: We evaluate GLM-5.3, Kimi-K3 and Qwen3.8 Max in Claude Code 2.1.207 (max reasoning effort, no web tools with `temperature=1.0`, `top_p=1.0`, `max_new_tokens=128000`). The reported results are single-run Pass@1 on 869 tasks under two timeout budgets: 2 hours and 6 hours, which are calculated as the API inference time rescaled by per-model tokens per second rate (per-model TPS sourced from Artificial Analysis; that is, we rescale GLM-5.3's results by 115 TPS, Kimi K3's results by 40 TPS and Qwen3.8 Max's results by 47 TPS), plus the non-API overhead. We also apply a domain whitelist (allowing only essential domains such as pypi.org and deb.debian.org for basic tool installation) to prevent the agent from cheating.
ExploitGym (2h / 6h) · GLM-5.3模型卡;2h / 6h · pass-at-1 · Tools allowed / max · Output limit 128000 tokens / Claude Code 2.1.207 · The card says reasoning_effort defaults to max and recommends keeping that default for benchmark reproduction. **ExploitGym**: We evaluate GLM-5.3, Kimi-K3 and Qwen3.8 Max in Claude Code 2.1.207 (max reasoning effort, no web tools with `temperature=1.0`, `top_p=1.0`, `max_new_tokens=128000`). The reported results are single-run Pass@1 on 869 tasks under two timeout budgets: 2 hours and 6 hours, which are calculated as the API inference time rescaled by per-model tokens per second rate (per-model TPS sourced from Artificial Analysis; that is, we rescale GLM-5.3's results by 115 TPS, Kimi K3's results by 40 TPS and Qwen3.8 Max's results by 47 TPS), plus the non-API overhead. We also apply a domain whitelist (allowing only essential domains such as pypi.org and deb.debian.org for basic tool installation) to prevent the agent from cheating.