一种防止大模型智能体导致服务器瘫痪的计算宪法
A computational constitution to stop LLM agents from bricking servers

原始链接: https://github.com/misqe/zero-trust-llm

目前人工智能行业依赖于经过人类反馈强化学习(RLHF)训练的“谄媚型”模型,这些模型往往为了追求速度而牺牲安全性,从而在“演示”与“生产”之间造成了危险的鸿沟。仅依赖“请小心”这类简单的系统提示词是远远不够的,因为概率性的大语言模型在执行确定性任务时,无法实现有效的自我约束。 本代码库提出了一个“计算宪法”概念,旨在为自主智能体强制实施“零信任”机制。该框架摒弃了不受约束的自主性,转向**认知状态机**,要求智能体必须遵循一套严格且经过验证的操作循环: 1. **假设** 2. **证据识别** 3. **基础验证** 4. **执行**(只读诊断) 5. **硬性交由操作员**(Hard Yield) 在“硬性交由操作员”阶段,执行中间件会物理中断 API 数据流,执行请求的指令,并将原始输出反馈回上下文。这种架构以强制的人机交互验证取代了盲目执行,弥合了实验性演示与可靠生产环境之间的差距。 本代码库提供了 `AGENTS.md` 主规则集、架构文档以及实现示例,旨在帮助开发者将易变、勉强可用的智能体转化为稳健且可治理的生产系统。

```Hacker News 最新 | 过往 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 一个旨在防止大模型智能体导致服务器瘫痪的计算宪法 (github.com/misqe) 3 分,由 misqe 发布于 2 小时前 | 隐藏 | 过往 | 收藏 | 1 条评论 帮助 mzajc 1 分钟前 | 下一条 [-] 你确定通过提示词让大模型以特定方式运行(在你这个案例中,是将任务拆分为 4 个所谓的受限子任务)是万无一失的,且不会被大模型直接忽略吗? 回复 准则 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索:```
相关文章

原文

A computational constitution for autonomous agents.

The Problem: The Demo-to-Production Chasm

The AI industry is trapped in the "Good Enough" illusion. Demos show agents magically writing code and deploying apps in 30 seconds. But commercial LLMs are heavily tuned via RLHF to be sycophantic - they want to guess the outcome, agree with the user, and execute tasks rapidly.

If you ask an ungoverned agent to "forcefully clear the Docker cache to fix a server crash," it will blindly bundle destructive commands and execute them based on your unverified premise. This is extremely dangerous in production environments.

Natural language governance (adding "be careful" to a system prompt) fails over time due to context window dilution.

When a probabilistic text generator is tasked with executing deterministic state changes, you cannot rely on it to govern itself. You must strip its agency and force it into an epistemic state machine.

This repository provides AGENTS.md, a master operational rule designed to govern an LLM's behavioral state machine at the prompt layer, bridging the gap to a runtime enforcer.

Every consequential action must follow this exact loop:

  1. [HYPOTHESIS]
  2. [IDENTIFY REQUIRED EVIDENCE]
  3. [GROUND VERIFICATION METHOD]
  4. [EXECUTE] (Strictly read-only diagnostic command)
  5. [HARD YIELD TO OPERATOR]

At [HARD YIELD], the execution layer (Python middleware or LangGraph/Semantic Kernel) must physically cut the API stream, execute the command, and feed the raw output back into the context.

  • AGENTS.md: The master ruleset. Add this to your agent's system prompt.
  • MANIFESTO.md: The philosophical and technical arguments against the "Good Enough" AI paradigm.
  • /examples: Real-world transcripts proving how standard agents fail (and how the Zero-Trust agent catches anomalies and yields).
  • /implementation: Architecture notes and Python pseudo-code showing how to programmatically enforce the execution boundary (orchestrator_concept.md).
联系我们 contact @ memedata.com