Gemini Pro · SWE-bench Verified · Verified subset (500 problems) · 历代记录

按发布时间查看 Gemini Pro 的 3 次发布在 SWE-bench Verified · Verified subset (500 problems) 的成绩、配置与披露。

仅为直接前代间的唯一、有效精确值计算差值,要求单位、指标与明确记录的配置和条件一致。

Gemini 2.5 Pro (GA)

· 本次发布有 2 条成绩记录,含 2 条精确值(可含历史记录)

Gemini 2.5 Pro (GA) · SWE-bench Verified 成绩与配置
指标与成绩配置与评测条件出处
Pass rate59.6%原文 59.6% · 单位 %
公布 未确认
Gemini 2.5 Pro stable API
设置
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Single agentic trace; internal harness; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; default sampling settings unless listed in the benchmark protocol.
在排名中查看此成绩
Pass rate67.2%原文 67.2% · 单位 %
公布 未确认
Gemini 2.5 Pro stable API
设置
AI Studio API model id gemini-2.5-pro; default sampling settings. Technical report states Gemini scores are pass@1 and single-attempt unless specified. Aider Polyglot averages 3 trials; SWE-bench uses the internal agentic harness; Vibe-Eval uses Gemini as judge.
未知项
Thinking budget and any non-default settings for individual evaluations are not stated except where noted in each protocol.
工具
工具条件未记录
推理预算
未记录
框架
未记录
协议
Multiple agentic traces sampled and reranked using Gemini's own judgement; temperature=1, top-p=0.99, top-k=1024.
条件
AI Studio API model gemini-2.5-pro; internal agentic harness.
在排名中查看此成绩

Gemini 3 Pro

· 本次发布有 1 条成绩记录,含 1 条精确值(可含历史记录)

与直接前代的记录缺少满足差值计算的同条件依据。

Gemini 3 Pro · SWE-bench Verified 成绩与配置
指标与成绩配置与评测条件出处
Pass rate76.2%原文 76.2% · 单位 %
公布 未确认
Gemini 3 Pro API default
设置
Gemini API model id gemini-3-pro-preview; default sampling settings. Gemini results are pass@1 and single-attempt unless specified; smaller benchmarks average multiple trials.
未知项
The evaluation table does not specify a thinking budget or inference-time compute setting.
工具
允许工具
推理预算
未记录
框架
未记录
协议
Single attempt; average over 10 runs; bash, file-operation and submit tools.
条件
Official provider methodology; different scaffolds and infrastructure mean results are not directly comparable.
在排名中查看此成绩

Gemini 3.1 Pro

· 本次发布有 1 条成绩记录,含 1 条精确值(可含历史记录)

与直接前代的记录缺少满足差值计算的同条件依据。

Gemini 3.1 Pro · SWE-bench Verified 成绩与配置
指标与成绩配置与评测条件出处
Pass rate80.6%原文 80.6% · 单位 %
公布 未确认
Gemini 3.1 Pro Thinking (High)
设置
Gemini API model id gemini-3.1-pro-preview; default sampling settings; evaluation table column specifies Thinking (High). All scores are pass@1 except where noted.
未知项
Exact thinking budget and sampling parameters are not listed.
工具
允许工具
推理预算
high
框架
未记录
协议
Single attempt; average over 10 runs; official provider method. Google adjusted the score by +0.6% after fixing three impossible official harness items.
条件
Gemini API model gemini-3.1-pro-preview; bash, file-operation and submit tools; not directly comparable to other providers' scaffolds.
在排名中查看此成绩