LLM Cookbook

Problem#

Learning to train language models from scratch means constantly cross-referencing papers, framework documentation, blog posts, and implementation guides — with the same core concepts (how RoPE works, how DPO differs from RLHF, how to train a tokenizer) explained slightly differently, and slightly incompletely, in a dozen scattered places. Most existing resources are either full linear courses (assuming you read chapter one before chapter ten) or one-off answers to a single question with no connection to the rest of the field.

LLM Cookbook is a structured, modular reference — short, self-contained articles you look up when you need them, in whatever order your actual problem demands.

Motivation#

The Cookbook started because I wanted a single reference I could continually expand while working on SLM From Scratch. I found myself repeatedly reading papers, documentation, blog posts, and implementation guides, but that knowledge was scattered across dozens of sources. Rather than continually searching for the same information, I wanted to build a structured reference that captured both the theoretical concepts and the practical engineering considerations in one place. Although it naturally complements SLM From Scratch, the long-term goal is for it to become a standalone reference for anyone building language models.

Architecture#

The guide is organized as one Markdown file per topic, grouped into eight categories under docs/<category>/<topic>.md: Fundamentals, Training, Datasets, Evaluation, Deployment, Serving, Optimization, and a set of end-to-end Projects. A small MkDocs Material site config builds this into a browsable reference.

This category-based layout replaced an earlier flat structure (all topics as files at the repo root) once the topic count grew past what a flat list could stay navigable at. Modules are designed to be individually self-contained — no module assumes you've read another one first — but a suggested learning path exists, and related modules link to each other for readers who want to go deeper on a referenced concept.

Runnable code is deliberately kept separate from the conceptual prose: two Jupyter notebooks (attention_mechanics.ipynb, lora_from_scratch.ipynb) and two example scripts live in their own directories, apart from the Markdown prose under docs/. That directory holds 34 Markdown files in total, counting every page — including docs/index.md, the site's own homepage, which isn't itself a topic module. Of those 34 files, 26 are the actual topic modules organized into the eight categories described below.

Core Features#

• 26 topic modules across 8 categories: transformers, attention, tokenization, and inference (Fundamentals); pretraining, scaling laws, SFT, LoRA, QLoRA, distillation, quantization, DPO, GRPO, and RLHF (Training); data filtering and dataset formats (Datasets); benchmarks, LLM-as-judge, and human evaluation (Evaluation); deployment and serving overviews; training-framework optimization; and three end-to-end walkthroughs (Projects).
• A glossary ("Definitions") covering Transformer, Tokenizer, Pretraining, Fine-tuning, LoRA/PEFT, Scaling laws, Knowledge distillation, RLHF/DPO/GRPO, and Evaluation/benchmarking.
• An explicit "What this guide is not" section, scoping it away from being a from-scratch implementation walkthrough (that's what SLM From Scratch is for) or a replacement for the primary sources it links to.
• Two hands-on Jupyter notebooks and two runnable example scripts for readers who want executable code alongside the conceptual prose.

Technical Decisions#

Reference, not tutorial. Most tutorials assume readers progress linearly from chapter one to chapter ten. That isn't how I actually learn or work. When implementing a feature, I usually need to answer one specific question — how RoPE works, how DPO differs from RLHF, or how a tokenizer should be trained — rather than follow an entire course again. I wanted every article to be independently useful while still fitting into a larger body of knowledge.

No tool bias. Maintaining tool neutrality required deliberate effort. Since most of my practical work uses the PyTorch ecosystem, it would have been very easy for the writing to gradually become PyTorch documentation in disguise. I wanted the conceptual articles to remain useful regardless of whether someone preferred PyTorch, JAX, TensorFlow, Unsloth, Axolotl, or another framework. Framework-specific examples belong in implementation notebooks, not in the conceptual explanations — confirmed in practice: only two of the 34 total Markdown files under docs/ mention PyTorch by name at all, and none mention TensorFlow or JAX.

Engineering Challenges#

The biggest challenge wasn't writing individual articles — it was designing an information architecture that would still make sense after hundreds of topics rather than a few dozen. The initial flat structure worked while the project was small, but as the number of modules grew it became increasingly difficult to navigate. The restructuring into categories was less about today's content and more about ensuring the project could continue to scale.

Screenshots#

Not applicable — LLM Cookbook is a prose reference, not a graphical application. There's no UI to screenshot; the closest visual artifact would be the built MkDocs site itself, which doesn't yet have a public deployment to capture.

Lessons Learned#

One lesson was that "self-contained" doesn't mean "isolated." Every article should answer its own question completely, but it should also clearly point readers toward related concepts instead of trying to teach everything at once. Good technical writing is often about deciding what not to include.

On code separation: I would keep it. Conceptual explanations and executable code serve different purposes. Mixing them often results in articles that are too abstract for implementation and code that lacks sufficient explanation. Keeping prose focused on concepts while placing runnable examples in notebooks has made both easier to maintain, and I expect that separation to become even more valuable as the Cookbook grows.

Current Status#

The Cookbook is not intended to be a "completed" project — it's an evolving knowledge base that grows alongside my understanding of language models, not a product with a defined finish line. As of now: 17 commits since the repository was created on 2026-07-23, most recently a category restructure (merged 2026-08-03) that moved from a flat file layout to the current docs/<category>/<topic>.md structure. No tagged releases exist, deliberately — there's no "1.0" this is building toward.

Tech Stack#

Markdown for the prose — 34 files total under docs/ (including the site index), 26 of which are the actual topic modules — built into a browsable site with MkDocs Material. The two hands-on notebooks and example scripts use plain PyTorch and Hugging Face Transformers — the one place the guide's tool-agnostic framing doesn't apply, since runnable code has to pick something concrete to run.

Related Content#

Practical Companion: SLM From Scratch is the hands-on implementation of the concepts covered here — training real language models from scratch on a single consumer GPU, applying the theory this guide documents. The Cookbook grew out of that project and remains its closest counterpart.