原始链接: https://epoch.ai/MirrorCode
**MirrorCode** 是与 METR 共同开发的一项全新基准测试，旨在评估 AI 模型在复杂、长周期软件工程任务中的表现。与专注于修复微小漏洞或实现小型功能的现有基准测试不同，MirrorCode 要求模型在无法获取原始源代码的情况下，从零开始重构整个程序，并需通过严格的端到端测试。 MirrorCode 的主要特点包括： * **规模化评估：** 提供充足的推理预算，允许模型在复杂任务上运行数天，从而模拟人类工程师的深度工作模式。 * **严谨的设计：** 任务在无互联网连接的沙盒环境中进行，以杜绝作弊行为。 * **验证可行性：** 表现优异的模型（如 Claude Opus）已成功重构了复杂的工具包（例如 16,000 行 Go 代码），且速度明显快于人类开发者。 尽管研究人员承认预训练数据污染可能会影响结果，但他们的“记忆筛选”机制表明，该基准测试能有效衡量真正的编程能力。MirrorCode 涵盖了不同计算领域的 25 个目标程序，其中 22 个已开源以鼓励进一步研究。这一举措标志着在评估 AI 处理持续性、高难度开发项目能力方面迈出了重要一步。
AI has made rapid progress on software engineering benchmarks in the past few years. However, most such benchmarks tend to focus on shorter tasks like fixing bugs or implementing individual features. MirrorCode is our benchmark, co-developed with METR, to test AI models on long-horizon coding tasks. In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI-generated solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.
Crucially, we provide a large enough inference budget to make a serious attempt at MirrorCode tasks. Many existing software engineering benchmarks limit inference spending to around $1–10, even when the task would take weeks for a human to complete. For example, one of the largest MirrorCode tasks cost $2,600 for a single run and involved AI working for 19 days without human intervention.
Reimplementing entire programs is extremely challenging for human software engineers. We believe a human engineer without AI would take months to solve the most complex MirrorCode tasks. However, MirrorCode tasks are also feasible; we know that there is enough information for the tasks to be fair.
We sandbox AI models, requiring them to conduct their work without access to the internet, without access to the original codebase, and with no way to cheat on the task. There are end-to-end tests that models never see while developing their code, so they cannot simply create a lookup table to mimic the original program's outputs.
AI can already solve long-horizon MirrorCode tasks, despite their difficulty. For example, Claude Opus 4.7 reimplemented gotree: a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands.1 We believe this same task would take a human engineer without AI assistance 2–17 weeks. Opus 4.7 solved it in 14 hours, costing $251.
One important caveat to these results is data contamination. Because MirrorCode tasks involve reimplementing open-source programs, AI models are likely to have seen the original codebases in pretraining. This might lead to inflated performance on the benchmark. However, AI successfully reimplemented several target programs that passed our memorization screen, and failed to reimplement programs where the screen showed evidence of memorization. This suggests that the results were not dominated by memorization, but we cannot rule out the possibility that memorization contributes to AI performance. Overall, we expect that the capabilities measured by MirrorCode would generalize to an unseen codebase. We discuss this further, along with more results and details on benchmark construction, in the paper.
MirrorCode is not fully solved. For our regularly updated leaderboard, we report MirrorCode (ML, +Private, 2L). This means we run the 15 target programs from the Medium and Large buckets, and drop the Small bucket. Each target program is evaluated in two implementation languages (generally Go and Ada) giving 30 tasks. We run each task three times, with a budget of 10 billion tokens per attempt.2
We release our scaffold and 22 of the 25 MirrorCode target programs (totaling 132 task instances across the six supported programming languages) as open-source, with the other three targets held out as a private test set.
This work was co-developed with METR and supported by a grant from METR. The authors of MirrorCode are Tom Adamczewski, David Owen, and David Rein. Florian Brand, Giles Edkins, Allen Hart, and Daniel O’Connell contributed additional target programs. Rasmus Faber-Espensen made crucial infrastructure improvements and gave advice on engineering