Research write-ups on training small language models from scratch: controlled experiments, what they showed, and what broke along the way.

Posts

• Oct 14, 2026

  GPT-2 (124M) to Gemma 3-Style (113M) language models, Part 2: KV caches, RoPE clocks, 4× context extension and attention visualization

• Oct 14, 2026

  GPT-2 (124M) to Gemma 3-Style (113M) language models, Part 1: Pretraining lessons from swapping one architecture component at a time