Case Studies · 案例研究

六大自进化系统深度分析

每个案例回答四个问题:它做了什么?怎么进化的?有什么证据?核心启示是什么?

Case 1 of 6 — Architecture Search

🧬 Darwin Godel Machine

项目卡 →

它做了什么?

An agent that rewrites its own Python source code to discover better agent architectures.

它怎么进化的?

Maintains an open-ended archive of agent designs. Each generation proposes code modifications, evaluates them on benchmarks, and keeps improvements in the archive for future generations to build on.

📊 证据

SWE-bench Verified
20.0% 50.0%
Polyglot Coding
14.2% 30.7%

💡 核心启示

Code-level self-modification with archival inheritance outperforms fixed-architecture approaches on real-world tasks.

Case 2 of 6 — Architecture Search

🏗️ Automated Design of Agentic Systems

项目卡 →

它做了什么?

A meta-agent that writes complete Python-coded agents, searching the space of agent designs automatically.

它怎么进化的?

Uses a "meta-agent" that proposes new agent architectures as Python code, evaluates them on tasks, and iterates. Discovered designs transfer across tasks and base models.

📊 证据

Transfer across tasks
Single-task Cross-task generalization confirmed
Design complexity
Manual prompt engineering Automated code search

💡 核心启示

Agent design space is searchable — meta-agents can discover non-obvious architectures that human designers miss.

Case 3 of 6 — Algorithm Discovery

💎 AlphaEvolve (DeepMind)

它做了什么?

Combines LLMs with evolutionary computation to discover novel algorithms for math, scheduling, and hardware design.

它怎么进化的?

LLM proposes code mutations -> automated evaluators score -> MAP-Elites maintains diverse population -> selected solutions survive. The paper reports a new matrix multiplication result in a historically studied task.

📊 证据

4×4 complex-valued matrix multiplication
49 scalar multiplications (Strassen setting) 48 scalar multiplications (reported after 56 years)
Data center scheduling
Standard heuristic Reported ~0.7% compute recovery in the paper context

💡 核心启示

LLM + evolution + strong evaluator can support algorithm discovery on tasks with executable scoring; claims should stay tied to the reported benchmark context.

Case 4 of 6 — Skill Library

🎮 Voyager (Minecraft Lifelong Learning)

项目卡 →

它做了什么?

An LLM-powered agent that plays Minecraft by building a reusable skill library through autonomous exploration.

它怎么进化的?

Three components: automatic curriculum (what to learn next), skill library (reusable code functions), and iterative prompting (fix failed skills). Skills compound — early discoveries enable later ones.

📊 证据

Unique items obtained
~50 (baseline) ~250 (3.6x)
Tech tree mastery
Partial Full diamond-tier progression

💡 核心启示

Skill libraries can create compound returns, but stale or mis-retrieved skills can also pollute future behavior.

Case 5 of 6 — Reflection

🪞 Reflexion: Language Agents with Verbal Reinforcement Learning

项目卡 →

它做了什么?

Agents that learn from failures by generating verbal reflections stored in episodic memory.

它怎么进化的?

After each failed attempt, the agent generates a textual "reflection" about what went wrong. Future attempts retrieve relevant reflections as additional context, avoiding repeated mistakes.

📊 证据

HumanEval pass@1
80% (GPT-4 baseline) 91%
AlfWorld success
77% 97%

💡 核心启示

Language can act as a medium of self-improvement when reflections are scoped, retrievable, and checked against task outcomes.

Case 6 of 6 — Self-Play

🏆 Self-Rewarding Language Models

它做了什么?

Language models that generate their own training rewards, creating a self-improvement loop without external annotation.

它怎么进化的?

The model generates candidate responses → judges them using its own reward model → trains on self-judged best responses via DPO → the improved model becomes a better judge → repeat.

📊 证据

Reward model quality
Iteration 1 Improves with each iteration (self-reinforcing)
Training data cost
Human annotation required Fully automated

💡 核心启示

The evaluator and the evaluated can be the same system — but only with careful design to prevent reward hacking.

Comparison

六大系统对比

系统进化维度进化环可变对象核心证据
DGMArchitecture SearchSearch + PopulationCodeSWE-bench 20→50%
ADASArchitecture SearchSearch + EvaluatorCodeCross-task transfer
AlphaEvolveAlgorithm DiscoveryPopulation + EvaluatorCode4×4 complex matmul 49→48, source-reported
VoyagerSkill LibraryReflection + MemorySkill, MemoryItems 3.6x
ReflexionReflectionReflectionMemoryHumanEval 91%
Self-RewardingSelf-PlaySearch + EvaluatorPolicyZero annotation
Synthesis

六个案例教会我们什么?

1. 代码是高杠杆的可变对象

六个案例中三个以代码为进化载体。修改代码可以改变系统行为,但必须由测试、回滚和迁移检查约束。

2. 评估器决定进化方向

AlphaEvolve 在可自动验证的数学/算法任务中报告结果,DGM 使用 SWE-bench 等评估面。评估器质量决定结果质量;本站需要继续区分 source-reported、reviewed 和 reproduced。

3. 记忆和技能创造复合回报

积累的经验可能产生复合回报,但也会引入过期记忆、错误归因和上下文污染。

4. 种群 > 单轨迹

维护多个候选方案并进行选择,比沿着单一方向优化更鲁棒。

5. 自我奖励有条件成立

如何防止评估者和被评估者是同一系统时的奖励作弊?

6. 真实环境是关键检验

从 Minecraft 到 SWE-bench:真实世界复杂性能够暴露 demo 和 benchmark 难以覆盖的失败模式。

Corpus funnel
数据漏斗:案例研究覆盖范围。
Star distribution
Star 分布:案例代表研究前沿。