Astra 6.2 和 Sol 6.2 在 Simple Bench 上表现不佳
Astra 6.2 and Sol 6.2 flop at Simple Bench

原始链接: https://simple-bench.com/

SimpleBench 是一个包含200多道关于时空推理、社交智能和陷阱题的多项选择文本基准。与大多数基准不同,非专业人类的表现优于顶尖大语言模型:一个小规模人类样本得分为83.7%,而 Claude Fable 得分为81.9%。结果表明,前沿模型的记忆知识和近似推理并不总是足以回答基础问题。

Hacker News 讨论 Simple Bench 结果:Astra 6.2 和 Sol 6.2 落后于 Anthropic 与 Google 的新模型,Fable 5.2 领先。评论者对基准回归是否重要有分歧:一些人不信任合成基准,认为实际适用性更重要,并指出 6.2 的速度和简洁;另一些人报告任务遵从性回归,模型会提前停止、走捷径或写出粗糙代码。多位用户称赞 Sol 6.2 Ultrafast/Cerebras 式速度以及 Neon/Gemini 模型,而另一些人认为 Opus/Fable 在复杂工作上更可靠。该基准的陷阱题风格和公开数据集受到批评,用户也在争论基准有效性与个人经验。总体而言,许多人认为 OpenAI 的 6.2 发布令人失望,但有些人重视其速度和特定优势。
相关文章

原文

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.

For a deeper dive into our results and our methods, check out the full technical report here.

Use all of these models on the Simple Bench app - LMcouncil.ai

联系我们 contact @ memedata.com