AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Single agentic trace; internal harness; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; default sampling settings unless listed in the benchmark protocol.
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Multiple agentic traces sampled and reranked using Gemini's own judgement; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; internal agentic harness.
Gemini 2.5 Pro (GA) · SWE-bench Verified · Verified subset (500 problems) · Scores and configurations
Gemini 2.5 Pro (GA) record for SWE-bench Verified · Verified subset (500 problems). 2 score records for this release, including 2 exact values (may include historical records). Includes 4 source locations.
Gemini 2.5 Pro (GA) · SWE-bench Verified scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Pass rate59.6%Reported 59.6% · Unit %
Published Unconfirmed
Gemini 2.5 Pro stable API
Settings
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
Unknowns
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
Tools
Tool access not recorded
Reasoning budget
not recorded
Framework
not recorded
Protocol
Single agentic trace; internal harness; temperature=1, top-p=0.99, top-k=1024.
Conditions
AI Studio API model gemini-2.5-pro; default sampling settings unless listed in the benchmark protocol.
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
Unknowns
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
Tools
Tool access not recorded
Reasoning budget
not recorded
Framework
not recorded
Protocol
Multiple agentic traces sampled and reranked using Gemini's own judgement; temperature=1, top-p=0.99, top-k=1024.
Conditions
AI Studio API model gemini-2.5-pro; internal agentic harness.