OpenArch – 现代大语言模型架构的 PyTorch 实现

OpenArch – 现代大语言模型架构的 PyTorch 实现
OpenArch – PyTorch implementations of modern LLM architectures

原始链接: https://github.com/anuj0456/OpenArch

该代码仓库提供了一系列具有教育意义的手写 PyTorch 实现，涵盖了现代大语言模型（LLM）架构。本项目灵感源自 Sebastian Raschka 的 LLM 架构画廊（LLM Architecture Gallery），与 `transformers` 等生产级库中的性能优化代码不同，本项目优先考虑代码的可读性和结构清晰度。每个模型都包含在各自的目录中，能够直观地展示架构细节，例如注意力机制（MHA、GQA、MLA）、归一化策略以及 MoE 路由等——这些细节在经过高度优化和分布式处理的代码库中往往难以察觉。目前，该项目已包含 Llama 3、DeepSeek R1 和 Qwen 3 等主流模型的可用实现，并计划覆盖画廊中的全部 72 种架构。作为一个学习资源，该仓库为那些希望从零开始理解现代大语言模型构建方式的学习者提供了理想的参考。作者欢迎社区贡献，包括新的模型实现、文档编写和测试，前提是这些贡献需秉持本项目清晰易懂的教育宗旨。所有代码均在 Apache License 2.0 许可下发布。

Python implementations of modern open-source LLM architectures — written from scratch, one model at a time.

This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery. Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery.

The goal is not to compete with transformers or other production libraries. The goal is clarity and learning: a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side.

Modern LLM architectures share a common skeleton but differ in dozens of small, important choices:

Attention: MHA, GQA, MQA, MLA, sliding-window, linear/DeltaNet hybrids
Normalization: pre-norm, post-norm, QK-Norm, sandwich norm, RMSNorm
Positional encodings: RoPE, NoPE, partial RoPE, YaRN
Decoder type: dense vs sparse MoE (with or without shared experts), hybrid Mamba/attention
Training-time tricks: Multi-token-prediction, latent experts, gated attention

Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading.

What's implemented (so far)

Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction.

Modality	Model	Status	Model Size	Normalization	Positional Encoding	Attention	Mixture of Experts
Text	GPT-2 XL	✅	1.5B	-	Absolute	Multihead Attention	No
	Llama 2	✅	7B	RMS Norm	RoPE	Multihead Attention	No
	Llama 3	✅	8B	RMS Norm	RoPE	Grouped Query Attention	No
	OLMo 2	✅	7B	RMS Norm & QK-Norm	RoPE	Multihead Attention	No
	DeepSeek R1	✅	671B	RMS Norm & QK-Norm	RoPE	Multihead Latent Attention	Yes
	Gemma 3	✅	27B	RMS Norm & QK-Norm	RoPE	Grouped Query Attention with Sliding Window	No
	Mistral 3	✅	24B	RMS Norm	RoPE	Grouped Query Attention with Sliding Window	No
	Llama 4 Maverick	✅	400B	RMS Norm	RoPE	Grouped Query Attention	Yes
	Qwen 3	✅	4B	RMS Norm & QK-Norm	RoPE	Grouped Query Attention	No
	Qwen 3	✅	30B-A3B	RMS Norm & QK-Norm	RoPE	Grouped Query Attention	Yes
	Kimi K2	✅	1T	RMS Norm	RoPE	Multihead Latent Attention	Yes
	GLM 4.5	✅	355B	RMS Norm & QK-Norm	RoPE	Grouped Query Attention & Multi-Token Prediction	Yes
	GPT-OSS	✅	20B	RMS Norm	RoPE	Grouped Query Attention with Sliding Window	Yes
	Grok-2.5	🚧	270B	RMS Norm	RoPE	Grouped Query Attention	Yes
Multimodal	PaliGemma	✅	3B	RMS Norm	RoPE	Multihead Attention	No
Multimodal	Qwen3	🚧	3B	RMS Norm	RoPE	Multihead Attention	No
Image	Dall-e	🚧	-	-	-	Transformer	-

The full target list mirrors the 72 architectures in the Architecture Gallery. Contributions toward any of them are welcome.

OpenArch/
├── text/
│   ├── gpt2/
│   │   ├── model.py
│   │   └── README.md
│   ├── llama3/
│   ├── qwen3/
|   ├── grok2.5/
│   └── deepseek_v3/
├── multimodal/
│   └── pali-gemma/
│       ├── model.py
│       └── README.md
├── README.md
└── requirements.txt

Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used.

I am actively looking for contributors. If you enjoy reading model papers, comparing config.json files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start.

Good first contributions:

Pick an unimplemented model from the gallery and add a model.py for it
Add a README.md for an existing model documenting its architectural choices
Add a forward-pass test that loads the official weights and matches outputs on a few tokens
Fix bugs, improve docstrings, or refactor shared components

Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first.

See CONTRIBUTING.md for more details.

This repository would not exist without the work of two outstanding educators:

Any errors in the implementations here are entirely my own.

This project is licensed under the Apache License 2.0 — see LICENSE for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics.

These implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a learning resource and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or transformers.

OpenArch – 现代大语言模型架构的 PyTorch 实现 OpenArch – PyTorch implementations of modern LLM architectures

What's implemented (so far)

OpenArch – 现代大语言模型架构的 PyTorch 实现
OpenArch – PyTorch implementations of modern LLM architectures