SLM From Scratch

Problem#

Understanding how language models are trained from papers and blog posts only goes so far — it tells you what works, not why the decisions in a real training pipeline matter. SLM From Scratch is a hands-on exploration of that gap: building a transformer, its training loop, and its data pipeline from scratch, and actually running it — on hardware anyone could plausibly own, not a GPU cluster.

Motivation#

SLM From Scratch exists because I wanted to move beyond understanding the theory of language models and actually experience the engineering involved in training one. Reading papers explains what works, but building the training pipeline yourself teaches you why those decisions matter. The project naturally became the practical companion to the LLM Cookbook — one explains the concepts, the other validates them through implementation.

The hardware constraint — a single RTX 5070 Ti (16GB VRAM) — was intentional, not incidental. Modern LLM research often assumes access to large GPU clusters, but I wanted to explore how much could realistically be learned on consumer hardware: what many enthusiasts, students, and independent developers can actually afford. Demonstrating that meaningful experimentation is possible within those constraints was part of the project's purpose from the beginning.

Architecture#

The project is structured as a 5-phase plan; Phase 1 — a working transformer, training loop, and generation pipeline — is complete:

• src/model.py — a GPT-style transformer built from scratch: causal self-attention, MLP blocks, transformer blocks, assembled into a full GPT model.
• data/prepare.py — tokenizes OpenWebText (~38GB raw, 8M documents) into train.bin/ val.bin using GPT-2's BPE tokenizer, yielding roughly 8.5B tokens.
• src/train.py — the training loop: cosine learning-rate schedule with warmup, AdamW (betas 0.9/0.95, weight decay 0.1), gradient accumulation, gradient clipping, mixed precision, torch.compile, Flash Attention via PyTorch's SDPA, and Weights & Biases logging.
• src/generate.py — top-k sampling from a trained checkpoint.

Show source

Two model configurations were trained end-to-end on this pipeline: a 50M-parameter model (6 layers, 6 heads, 384-dim embeddings) and a 77M-parameter model (8 layers, 8 heads, 512-dim embeddings), each for 5,000 steps (~80M tokens) on the RTX 5070 Ti. Phases 2 through 5 — custom tokenizer training, a scaling-law experiment grid, knowledge distillation from a larger teacher model, and a formal evaluation harness — are documented as a plan, not yet built.

Core Features#

• A transformer built from first principles: attention, MLP, and transformer blocks, not a library abstraction.
• A full training loop with the practical details that make real training work: mixed precision, torch.compile, Flash Attention/SDPA, gradient accumulation, gradient clipping, checkpointing, and live metric logging to Weights & Biases.
• An OpenWebText data pipeline, tokenized with GPT-2's BPE tokenizer into a multi-billion-token training set.
• Text generation via top-k sampling from any trained checkpoint.
• Two trained model configurations (50M and 77M parameters) with documented, comparable results.

Technical Decisions#

The single-GPU constraint shaped the entire project, deliberately. Rather than renting cloud compute to train larger models faster, the project stayed on one RTX 5070 Ti throughout — the point wasn't to train the biggest model possible, it was to find out what's actually learnable about transformer training within hardware most people could realistically own.

The project is also structured as five explicit phases rather than one open-ended effort: build the core pipeline first (Phase 1), then use it to run controlled experiments (scaling laws, distillation, evaluation) once that foundation is solid — rather than trying to explore all of it in parallel before any of it worked reliably.

Engineering Challenges#

Getting the training pipeline working reliably was the first milestone, and it had several parts that all had to work together before any experiment was meaningful: mixed precision, Flash Attention/SDPA, torch.compile, gradient accumulation, checkpointing, and evaluation.

Once that foundation existed, the more interesting challenge became interpreting the experimental results rather than producing them. The 77M model underperforming the 50M model at equal token count — despite having more capacity — is a genuine finding, not a bug: it reinforced that simply increasing parameter count doesn't help if the training budget (tokens seen) stays fixed. That result made scaling-law discussions concrete in a way reading about them doesn't.

Phase 1 stopping there was a deliberate scope decision, not the project running out of momentum: it successfully demonstrated the full training pipeline and produced enough experimental evidence to validate what the phase was meant to prove. Rather than immediately training progressively larger models, the plan is to consolidate what Phase 1 showed before expanding scope — the project is meant to evolve iteratively, not become one continuous, open-ended training run.

Screenshots#

Lessons Learned#

The biggest lesson is that training language models is far less mysterious than it initially appears. Once the infrastructure is in place, the process becomes a series of engineering trade-offs rather than magic. Reading papers explains the algorithms, but implementing the pipeline teaches the practical considerations — memory constraints, checkpointing, optimizer behavior, data quality, debugging numerical issues — that are difficult to appreciate without running the experiments yourself.

Before running experiments, scaling laws felt like theoretical guidance. After training models myself, they became concrete engineering constraints. Seeing a larger model underperform because it was undertrained reinforced that parameters, tokens, compute budget, and optimization all need to be considered together — simply increasing model size isn't a free improvement, it changes the entire training budget.

Current Status#

Phase 1 of a 5-phase plan is complete and consolidated: a working transformer, training loop, data pipeline, and generation script, with two trained model configurations and documented results. Phases 2 through 5 (custom tokenizer training, a scaling-law experiment grid, knowledge distillation, and a formal evaluation harness) remain a documented plan rather than built code — this is an intentional pause to consolidate what Phase 1 showed, not an abandoned project. The plan is to release this alongside the LLM Cookbook, and to eventually fold in related work from three connected personal projects — dpo-from-scratch (preference alignment), llm-eval-arena (evaluation), and domain-llm (domain specialization) — into that same combined reference.

Tech Stack#

Python, PyTorch 2.6 with torch.compile and SDPA-based Flash Attention, Hugging Face transformers/tokenizers/datasets for the data pipeline, and Weights & Biases for experiment tracking.

Related Content#

Background Reading: LLM Cookbook documents the theory and engineering concepts used throughout this project — tokenization, scaling laws, distillation, and evaluation are all covered there in depth. This project is the hands-on validation of those ideas.