AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Single agentic trace; internal harness; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; default sampling settings unless listed in the benchmark protocol.
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Multiple agentic traces sampled and reranked using Gemini's own judgement; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; internal agentic harness.
Gemini API model id gemini-3-pro-preview; default sampling settings. Gemini results are pass@1 and single-attempt unless specified; smaller benchmarks average multiple trials.
未知项
The evaluation table does not specify a thinking budget or inference-time compute setting.
工具
允许工具
推理预算
未记录
框架
未记录
协议
Single attempt; average over 10 runs; bash, file-operation and submit tools.
条件
Official provider methodology; different scaffolds and infrastructure mean results are not directly comparable.
Gemini API model id gemini-3.1-pro-preview; default sampling settings; evaluation table column specifies Thinking (High). All scores are pass@1 except where noted.
未知项
Exact thinking budget and sampling parameters are not listed.
工具
允许工具
推理预算
high
框架
未记录
协议
Single attempt; average over 10 runs; official provider method. Google adjusted the score by +0.6% after fixing three impossible official harness items.
条件
Gemini API model gemini-3.1-pro-preview; bash, file-operation and submit tools; not directly comparable to other providers' scaffolds.
Gemini Pro · SWE-bench Verified · Verified subset (500 problems) · Release history
View scores, configurations, and disclosures for SWE-bench Verified · Verified subset (500 problems) across 3 Gemini Pro releases, ordered by release date.
Differences are calculated only for unique, active, exact values between direct predecessors when the unit, metric, and explicitly recorded configuration and conditions match.
· 2 score records for this release, including 2 exact values (may include historical records)
Gemini 2.5 Pro (GA) · SWE-bench Verified scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Pass rate59.6%Reported 59.6% · Unit %
Published Unconfirmed
Gemini 2.5 Pro stable API
Settings
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
Unknowns
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
Tools
Tool access not recorded
Reasoning budget
not recorded
Framework
not recorded
Protocol
Single agentic trace; internal harness; temperature=1, top-p=0.99, top-k=1024.
Conditions
AI Studio API model gemini-2.5-pro; default sampling settings unless listed in the benchmark protocol.
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
Unknowns
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
Tools
Tool access not recorded
Reasoning budget
not recorded
Framework
not recorded
Protocol
Multiple agentic traces sampled and reranked using Gemini's own judgement; temperature=1, top-p=0.99, top-k=1024.
Conditions
AI Studio API model gemini-2.5-pro; internal agentic harness.
· 1 score record for this release, including 1 exact value (may include historical records)
Records for the direct predecessor do not provide matching conditions for a valid difference.
Gemini 3 Pro · SWE-bench Verified scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Pass rate76.2%Reported 76.2% · Unit %
Published Unconfirmed
Gemini 3 Pro API default
Settings
Gemini API model id gemini-3-pro-preview; default sampling settings. Gemini results are pass@1 and single-attempt unless specified; smaller benchmarks average multiple trials.
Unknowns
The evaluation table does not specify a thinking budget or inference-time compute setting.
Tools
Tools allowed
Reasoning budget
not recorded
Framework
not recorded
Protocol
Single attempt; average over 10 runs; bash, file-operation and submit tools.
Conditions
Official provider methodology; different scaffolds and infrastructure mean results are not directly comparable.
· 1 score record for this release, including 1 exact value (may include historical records)
Records for the direct predecessor do not provide matching conditions for a valid difference.
Gemini 3.1 Pro · SWE-bench Verified scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Pass rate80.6%Reported 80.6% · Unit %
Published Unconfirmed
Gemini 3.1 Pro Thinking (High)
Settings
Gemini API model id gemini-3.1-pro-preview; default sampling settings; evaluation table column specifies Thinking (High). All scores are pass@1 except where noted.
Unknowns
Exact thinking budget and sampling parameters are not listed.
Tools
Tools allowed
Reasoning budget
high
Framework
not recorded
Protocol
Single attempt; average over 10 runs; official provider method. Google adjusted the score by +0.6% after fixing three impossible official harness items.
Conditions
Gemini API model gemini-3.1-pro-preview; bash, file-operation and submit tools; not directly comparable to other providers' scaffolds.