Correctness
Does the evolved artifact actually work?
Primary
Evaluation / 评估
Benchmark 对比与评估体系
自进化系统的评估不能只看 single-number accuracy。本页汇总四大领域的 benchmark 对比，并提出六维评估框架。
4评估领域
17+Benchmark
6评估维度
11自进化方法
Code Generation
Code Generation
| Benchmark | 规模 | 指标 | 最优自进化系统 | 基线 |
|---|---|---|---|---|
| HumanEval | 164 | pass@k | Reflexion (91%) | GPT-4 zero-shot (80.1%) |
| MBPP | 974 | pass@k | Self-Refine | Base model |
| DS-1000 | 1000 | pass@1 | SelfEvolve (49.4→57.2) | GPT-4 (49.4%) |
| SWE-bench | 2294 | % resolved | DGM (20→50%) | SWE-agent (20%) |
| LiveCodeBench | 800+ | pass@1 | ReVeal (36.9→42.4) | Base (36.9%) |
Mathematical Reasoning
Mathematical Reasoning
| Benchmark | 规模 | 指标 | 最优自进化系统 | 基线 |
|---|---|---|---|---|
| GSM8K | 8500 | % accuracy | SPIN Zephyr (26.8→39.0) | Zephyr-7B (26.8%) |
| MATH | 5000 | % accuracy | Agent Symbolic Learning (60.7%) | GPT-4 (~50%) |
| MiniF2F | 488 | % proved | AlphaProof | Prior SOTA |
| IMO 2024 | 6 | /6 solved | AlphaProof+AlphaGeometry (4/6) | Human gold (6/6) |
Agent Tasks
Agent Tasks
| Benchmark | 规模 | 指标 | 最优自进化系统 | 基线 |
|---|---|---|---|---|
| ALFWorld | 134 | % success | Reflexion (97%) | ReAct (75%) |
| WebArena | 812 | % success | GPT-4 (~14.4%) | Random (~1%) |
| HotPotQA | 113K | F1/EM | Reflexion | CoT baseline |
| Voyager (Minecraft) | open-ended | unique items | Voyager (3.3x) | AutoGPT (1x) |
Open-Ended Discovery
Open-Ended Discovery
| Benchmark | 规模 | 指标 | 最优自进化系统 | 基线 |
|---|---|---|---|---|
| Matrix Multiplication | 4×4 complex | # multiplications | AlphaEvolve (48 mul) | Strassen (56, 1969) |
| Cap Set | n=6,8,... | cap size | FunSearch (new records) | Prior SOTA |
| Borg Scheduling | Google cluster | % compute recovery | AlphaEvolve (+0.7%) | Hand-tuned heuristics |
| AI Scientist v2 | paper generation | review score | AI Scientist v2 (6.33) | Average workshop paper (~5) |
Framework
六维评估框架
Efficiency
What did the improvement cost (tokens, compute, time)?
Required
Transfer
Does improvement generalize to unseen tasks?
Critical
Robustness
Does performance hold across distributions?
Required
Safety
Does the change introduce new risks?
Mandatory
Cost
Total resource consumption per improvement unit
Required
每个维度同时适用于最终产物和进化过程本身。详见论文 Chapter 5。
Protocol
八点评估协议
1
Define Evolving Object
明确声明什么在被修改（prompt、code、architecture、weights）
2
Freeze Evaluator
评估器必须在实验开始前冻结，不能随进化过程变化
3
Report Adaptation Budget
报告总 token、计算量、时间、API 调用次数
4
Train/Val/Test Separation
进化用数据和最终评估数据必须分离
5
Measure Regressions
不只报告改进，还要报告是否有任务变差
6
Evaluate Transfer & Robustness
在新任务、新环境、新模型上验证迁移能力
7
Publish Process Traces
发布完整进化轨迹，包括失败候选
8
Normalize by Cost
所有改进必须按成本归一化：每美元/每 token 的收益