There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4, a 26-billion-parameter open-weights mixture-of-experts model, at about five tokens per second. Reading speed.

Anybody can rent a GPU. It’s harder to take a modern MoE model and a dead enterprise box and make them meet in the middle, and that gap is the whole reason I’m writing this up. “Good with AI” has quietly come to mean “pays for a subscription.” I think the real skill is different: knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct. So rather than claim we’re good at this, here’s a worked example, on hardware that had no business cooperating.

The post that started it

A couple of weeks ago a piece called “A 10 year old Xeon is all you need” made the rounds on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik_llama.cpp and about 25 carefully chosen flags. It’s a great read, and it leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering.

“I have a Xeon too,” I thought. Several, in fact. So I tried it. It didn’t run.

What an AI agent is actually good for

The build died on startup. I handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The author’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the generation Intel calls “v2.” The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn’t ship until Haswell, the “v3” generation, in 2014. My CPUs are older than the instructions the code was written against. The optimized paths weren’t there to execute.

So I asked the obvious follow-up: can we make it run anyway? I’d already taken a first swing with a free model that got close but couldn’t land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off, reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren’t there.

This is the part I care about. This didn’t come from typing “fix it” once and getting a working patch back. Somebody had to read another person’s performance-critical C++, work out why a kernel wasn’t valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. Claude did that work. My job was narrower: run the right experiments and recognize when the output was finally correct. I came away impressed.

The result

Gemma 4’s 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model’s architecture existed. The original write-up never published a tokens-per-second figure, just “reading speed,” so here’s the concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free.

Proof it runs: Gemma 4 26B answering on the basement box, CPU-only.

The patch is up as ikawrakow/ik_llama.cpp#2138 if you want the exact diff — still open and awaiting maintainer review as I write this, so run it from the branch for now. The hope is that anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn’t make sense.

For the people who want the actual bug

Full disclosure before I go further. I’m not a C++ programmer. I can read a stack trace and I know my way around a build system, but I did not hand-write kernel fallbacks for a quantized matmul engine, and I won’t pretend I did. What I did was drive. I ran the experiments, read the output, asked the next question, and knew what “correct” had to look like. The diagnosis and the patch came from the Claude instance running on the server itself. I asked it to write up what it fixed, and the rest of this section is that summary, lightly edited. If you came here from Hacker News for the real teardown, this part’s for you.

What was actually broken

The engine we needed was ik_llama.cpp, ikawrakow’s fork of llama.cpp that adds the optimizations Gemma 4’s MoE inference depends on. It assumes AVX2 as its floor. The Xeon E5-2690 v2 in this box has AVX1 but not AVX2. Turn GGML_USE_IQK_MULMAT off at build time and most of the codebase respects it: the fast paths compile out, and the model falls back to plain scalar/SSE math. That’s fine for a normal Q8_0 matmul.

Two graph ops are the exception. The Gemma 4 MoE feed-forward network emits MOE_FUSED_UP_GATE (a per-expert gate+up matmul fused with SwiGLU) and FUSED_UP_GATE (its dense analog). Both are #if-gated on GGML_USE_IQK_MULMAT inside the compute dispatcher, but the graph builder still emits them unconditionally. On this build the dispatcher’s switch had no case for those op enums, so they fell through to the default, and the destination tensors for every expert FFN silently never got computed. Gemma 4 26B has 30 layers by 8 active experts per token, so every forward pass consumed roughly 240 tensors of whatever happened to be sitting in that memory buffer already.

The symptom was fluent-looking multilingual gibberish. Token IDs spread uniformly across the 262K vocabulary, the model equally happy to emit Thai script, Korean, <unused> sentinels, or English fragments. Deterministic at temperature 0, byte-identical between single- and multi-threaded runs, no NaNs anywhere. Just a hidden state getting shoved by a large constant every layer until the final softmax went flat.

That determinism is what cracked it. Claude instrumented the raw logits before sampling, printing the top-5 tokens plus range, mean, and NaN count. The numbers gave it away: a mean logit of +16 for the first predicted token when it should sit near zero, and about 80% of the vocabulary at positive logits. Random corruption doesn’t look like that. A bias that clean only happens when a big chunk of the hidden state is uninitialized memory that happens to hold small positive floats.

The fix

Three commits on top of the fork’s main.

1. Compile fixes. The scalar #else branches for quantize_row_q8_0_x4 and quantize_row_q8_1_x4_T in iqk_quantize.cpp weren’t actually scalar. They still referenced hsum_i32_8 and other AVX2 helpers. Those got rewritten as portable scalar loops, with #if GGML_USE_IQK_MULMAT guards added around a handful of stray IQK calls leaking through ggml.c and ggml-quants.c, plus a missing include so iqk_cpu_ops.cpp compiles standalone. Without these, the fork won’t build at all on non-AVX2 hardware.

2. The runtime bug. Rather than touch the dispatcher, the fix makes the graph builder emit ops that do have compute paths on this build. In ggml_moe_up_gate, when GGML_USE_IQK_MULMAT is off: if the weight is the combined up_gate_exps tensor (shape [n_embd, 2*n_ff, n_experts], gate in the first half, up in the second), split it into two ggml_view_3d slices, run two separate ggml_mul_mat_id calls, and combine them with ggml_fused_mul_unary(gate, up, SILU). If gate and up are already separate weights, skip the split and do the same two mul-mat-IDs plus the fused mul-unary. ggml_fused_up_gate, the dense version used in non-MoE layers, gets the same treatment. Every op involved already has a working non-IQK implementation (mul_mat_id is stock ggml, and fused_mul_unary does the SILU-and-multiply in one pass). The whole change sits behind #if !GGML_USE_IQK_MULMAT, so an AVX2 build stays bit-identical to what it was before.

3. CI stubs. The #else stub sections of the iqk sources had drifted out of sync with iqk_mul_mat.h, so ci/run.sh couldn’t even build on non-AVX2 hardware: a missing <cstdint>, stubs with the wrong signatures (an extra leading parameter here, a missing sinks there), and no stubs at all for a couple of functions, which meant undefined references at link time. Boring work, but without it nobody on this hardware can run the test suite.

The fallback costs something, two separate matmul-IDs instead of one fused kernel, but this CPU is memory-bandwidth-bound anyway, and the fused kernel was AVX2-only, so we weren’t giving anything up. End to end we get about 5.2 tok/s decode and ~16 tok/s prompt-eval on a 26B-A4B MoE.

One more gotcha. --run-time-repack reorders quantized weights into an AVX2-only interleaved layout (Q8_0_R8) at startup, which garbles output on AVX1 the same way. That’s a separate bug, and the patch doesn’t try to fix it. The run script just drops the flag.

The instruction-set mismatch was easy to spot. The silent fall-through was not. Reading the code kept clearing the obvious suspects: the RMSNorm helpers looked correct, the AVX1 fallback in ggml_vec_dot_q8_0_q8_0 looked correct, and a bit-identical single-thread run ruled out threading. Only after instrumenting the logits, and seeing the mean pinned at +16 with every long-tail token roughly tied, did the search narrow to “a big chunk of the residual stream is uninitialized.” Grepping for #if GGML_USE_IQK_MULMAT in the dispatcher turned up the two missing cases about a minute later.

Reproduce it

If you have a pre-AVX2 box and want to try this:

• Hardware: dual Xeon E5-2690 v2 (Ivy Bridge, AVX1, no AVX2), DDR3, no GPU.
• Build: compile ik_llama.cpp from the branch above with GGML_USE_IQK_MULMAT off. The compile fixes are what let it build at all on non-AVX2 hardware.
• Model: Gemma 4 26B-A4B, Q8_0.
• Run: use the usual ik_llama.cpp CPU flags, but drop --run-time-repack (it reorders weights into an AVX2-only layout and re-garbles the output on AVX1).

That is the whole recipe: about 5 tok/s decode, CPU-only, no GPU anywhere in the box.

Why this is on the company blog

The subscription is the easy part. The rest is the willingness to open the hood, read a stranger’s code, and keep asking until a thirteen-year-old CPU does something it was never meant to. That’s the same work a fifteen-year-old Rails app needs, or a database nobody left on the team still understands: someone who’ll dig until they find where the leverage is, and what the tool won’t tell you on its own.

If you have pre-AVX2 iron gathering dust and try the branch, I’d love to hear what it does on your silicon — how far down the CPU generations does this go? The PR thread is the right place for bug reports. And if the thing you’re nursing along is a fifteen-year-old Rails app rather than a thirteen-year-old Xeon, that’s what we do for a living.

A couple of pointers for the curious. The server itself cost under $300; here’s the math on why a basement box beats $1,500 a month of cloud. And getting a screaming enterprise appliance quiet and bootable in the first place was its own project, written up here for people who enjoy that sort of thing.