超过 30% 的 ArXiv 新提交论文看起来是由 AI 撰写的。
How we measured AI writing across arXiv, and where the measurement breaks

原始链接: https://unslop.run/blog/measuring-ai-writing-on-arxiv

一项近期针对 12,750 篇 arXiv 论文的研究表明,截至 2026 年中期,约 32% 的新提交论文读起来像是机器撰写的。为确保准确性,研究人员利用 ChatGPT 问世前(2021-2022 年)的论文对检测器进行了校准,并将误报率设定在 0.4% 的阈值。这一“基准线”考虑了学术写作中的自然差异,以确保观察到的 AI 风格文本的增长并非仅仅是统计上的误差。 数据显示,在 ChatGPT 发布后,这一趋势呈现显著上升,且不同领域的采用率差异巨大。计算机科学领域有 65% 的论文被标记为机器撰写,而数学领域则仅为 0.7% 左右。 然而,研究指出了以下关键局限性: * **检测盲区:** 像数学这样高度依赖符号的领域,得分较低可能是由于检测器的“盲点”,而非缺乏 AI 使用。 * **结果解读:** 结果反映的是“机器化”写作的普遍性(包括大量 AI 辅助编辑),而非完全由 AI 撰写的证据。 * **低估倾向:** 由于检测器无法识别所有模型或提示词组合,报告的数字应被视为 AI 集成程度的下限。 研究人员得出结论,尽管学术界对 AI 的应用已相当广泛,但这些评分反映的是总体趋势,而非针对个人的指控。

最近一项针对超过 12,000 篇 arXiv 论文的分析显示,表现出人工智能生成文本特征的论文数量急剧增加。到 2026 年 1 月,约 39% 的提交论文(计算机科学类论文中占比高达 65%)被标记为 AI 撰写,而在 ChatGPT 发布之前,这一比例几乎为零。 研究人员之间的讨论凸显了对这一转变影响的巨大分歧: * **实用性视角:** 一些人认为,如果 AI 工具在不损害研究结论的前提下提高了科研交流的效率,那么作者的创作方式并不重要。 * **“垃圾内容”(Slop)的风险:** 批评者认为,大语言模型破坏了基本的质量信号机制。由于 AI 能够生成文笔流畅但内容空洞的文本,读者将更难区分严谨的研究和“垃圾内容”。 * **信任与诚信:** 一个主要担忧是“幻觉”现象的增加,例如伪造引文。由于 AI 能够大规模生产科学内容,这可能会导致科研社区充斥着低质量、未经验证或完全编造的信息,使得同行评审人员越来越难以进行维护科学标准所必需的“审慎阅读”。
相关文章

原文

We scored the full text of 12,750 arXiv papers and found that about a third of new ones read as machine-written. Here is the method, the results, and an honest account of the limitations.

Share of new arXiv papers flagged as machine-written, 2021 to 2026, at a threshold calibrated so pre-ChatGPT papers flag at 0.4%. The eight slate points are the pre-LLM control months; the band is a bootstrap 95% interval.
Share of new arXiv papers flagged as machine-written, 2021 to 2026, at a threshold calibrated so pre-ChatGPT papers flag at 0.4%. The eight slate points are the pre-LLM control months; the band is a bootstrap 95% interval.
Share of papers flagged as machine-written by field, over the 12 months to July 2026 (about 300 papers per field).
Share of papers flagged as machine-written by field, over the 12 months to July 2026 (about 300 papers per field).

A false-positive floor

There is a genre of headline that says "N% of X is now AI," and most are not worth reading, because the detector behind the number also flags some share of genuine human writing. If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned.

So we built the study around that objection. Our detector, described here, is calibrated for academic writing; at a 0.4% false-positive rate it clears 99.6% of genuine pre-LLM scientific text and recovers 85% of AI academic text. We made that false-positive rate the anchor. We took papers submitted in 2021 and 2022, before ChatGPT, treated them as ground-truth human, and set the flag threshold so that exactly 0.4% of them trip it. That line is the floor. Every number we report is a share of papers above a threshold where genuine pre-LLM writing sits, by construction, at 0.4%. The pre-ChatGPT years then act as a built-in control: if the rise were an artifact of the detector, 2021 and 2022 would flag as high as 2026. The first figure shows they do not.

What we measured

We sampled ten field groups, roughly 25 papers per field per month, from January 2023 to July 2026, plus eight control months across 2021 and 2022, for 12,750 papers in total. For each one we pulled the version-1 PDF, so a paper revised in 2026 cannot leak modern text back into its 2023 slot. We scored the full body text instead of the abstract, because abstracts understate the signal: we have seen the same paper score under 20% on its abstract and over 70% on its body. Every reported figure carries a bootstrap 95% confidence interval.

Results

The flagged share is flat at 0.4% through 2021 and 2022, lifts off within months of ChatGPT, and climbs in two waves to about 32% over the most recent complete quarter, peaking near 39% in early 2026. The spread across fields is large, and it is the table and the second figure that carry it. The values below are each field's flagged share over the 12 months to July 2026, alongside its pre-LLM control level.

Field groupPre-LLM controlRecent flagged share95% CI
Computer science0.2%65.0%[59.3, 70.3]
Quantitative biology3.5%56.3%[51.0, 61.7]
Electrical eng. & systems1.7%51.3%[46.0, 57.0]
Economics & finance2.5%47.0%[41.3, 52.7]
Applied physics1.3%34.0%[29.0, 39.7]
Statistics1.8%31.3%[26.0, 36.7]
Condensed matter0.0%24.0%[19.3, 29.0]
High-energy physics0.5%14.0%[10.0, 18.0]
Astrophysics0.0%10.7%[7.3, 14.3]
Mathematics0.0%0.7%[0.0, 1.7]

Computer science leads at about 65%. Mathematics is lowest, near 0.7%, and the limitations section explains why its low value is hard to interpret. The control column is each field's 2021 to 2022 flag rate averaged over three sensitivity settings; the fields that rise most are not the ones with the highest pre-LLM control level, so an elevated starting point does not explain the rise.

Limitations

Control sample size. Each field's pre-ChatGPT control is 200 papers. At a 0.4% flag rate only eight papers flag across the entire 2,000-paper control, spread thinly over ten fields, so a single-threshold per-field control rate is coarse. The pooled floor is well estimated and is what the study is anchored to, but the per-field control levels are only approximate, and a larger control would not fix this: pinning a fraction-of-a-percent rate per field would require thousands of control papers per field that pre-2023 arXiv volume does not contain.

A low score can indicate low adoption or a detector blind spot. Mathematics is the clearest case. Mathematics papers are dominated by notation and theorem-proof structure, and once equations and references are removed the remaining prose is sparse and unlike the scientific English the detector was trained on. A mathematics paper drafted with heavy model assistance may score low because its prose is out of distribution for the detector, so a low score in mathematics is weak evidence that a human wrote the paper. The result is consistent with two very different explanations, lower adoption or reduced detector sensitivity in that register, and this data cannot separate them. The fields with the strongest in-distribution assumption, the prose-heavy ones, are also the ones that rise most, so this confound does not account for the aggregate trend. But in the low-scoring fields the ranking should be read as a lower bound on adoption.

Detector coverage. The detector is more sensitive to some generators than others, and we cannot evaluate it against the exact, private mixture of models and prompts that authors actually use. Incomplete coverage lowers the flag rate, so the reported prevalence is a lower bound: the true share is at least what we measured. The detector write-up reports the per-generator performance.

A flag is not authorship. The detector estimates whether text reads as machine-written, at a calibrated probability with a known error rate. It cannot separate a lightly-edited document from a wholly-generated one, and a single score is never grounds to accuse a specific person. We report the prevalence of machine-like writing, which includes heavy AI-assisted editing.

Try it

The detector is cheap to run and we make no money from it. You can try it for free on any arXiv paper here, and on your own text here.

联系我们 contact @ memedata.com