Build Your Own LLM

First Green Test

Text normalization prevents artificial vocabulary inflation from casing and spacing artifacts, so downstream counts represent real language patterns.

Counting What the Model Sees

Token-frequency counting is the first measurable signal a language model can extract from data.

Random Choice for Generation

Weighted sampling turns static token probabilities into diverse generation behavior.

Unigram Baseline Generator

A unigram baseline closes the full loop from training counts to generated output.

Tokenize and Vocabulary

Vocabulary mapping is the numeric interface between text and model computation.

Build Bigram Counts

Bigram counts capture first-order local order by tracking which token follows which.

Bigram Probabilities and Predict

Normalization turns raw transition counts into valid next-token probability distributions.

Generate with Bigram

Bigram generation is the first model in this curriculum that conditions on immediate context.

Make Context Target Examples

Supervised next-token training requires explicit (context, target) pairs from token streams.

One-Hot and Linear Logits

A linear logit layer is the simplest trainable scorer from token representation to vocabulary scores.

Softmax and Cross-Entropy

Transformer training optimizes next-token probabilities via softmax outputs and cross-entropy loss.

Gradient Step That Reduces Loss

Gradient updates are the mechanism that turns prediction errors into parameter improvements.

Embedding Lookup

Embedding lookup replaces sparse one-hot inputs with dense trainable token representations.

Combine Context Vectors

A language head needs one context representation per example before scoring next-token logits.

MLP Language Head

An MLP head adds non-linearity so the predictor can model richer token interactions.

Train Neural Language Model

This lesson connects embeddings, context composition, logits, loss, and updates in one reproducible loop.

Query, Key, Value Intuition

In Attention Is All You Need, attention output is a weighted sum of value vectors using query-key compatibility weights.

Attention Scores and Weights

Scaled dot-product attention computes QK^T / sqrt(dKey) and applies row-wise softmax to produce attention weights.

Weighted Sum of Values

The paper's attention output multiplies weights by value vectors to produce contextualized representations.

Causal Mask (No Peeking)

Decoder self-attention must block future-token access so position i attends only to positions <= i.

Multi-Head Attention

Section 3.2.2 projects Q, K, and V into multiple subspaces so different heads can attend to different patterns.

Feed-Forward Layer

The Transformer applies a position-wise FFN: max(0, xW1 + b1)W2 + b2 after attention mixing.

Residual and LayerNorm

Residual paths preserve signal, and normalization stabilizes scale across stacked sublayers.

Full Transformer Block Forward Pass

A full block composes attention, residual/normalization, FFN, and residual/normalization in a fixed order.

Positional Information

Section 3.5 adds positional information because attention alone has no recurrence or convolutional order signal.

Stack Blocks Into MiniGPT

Stacking blocks creates the full forward path from embeddings to vocabulary logits.

Batching and Train Step

Batching makes optimization efficient and keeps tensor contracts consistent across updates.

Validation and Checkpoints

Validation estimates generalization, and checkpoints preserve exact model state for reproducible comparison.

Temperature Sampling

Temperature rescales logits during sampling to control distribution sharpness without retraining weights.

Top-k Sampling

Top-k sampling restricts decoding to the highest-scoring candidates before drawing a token.

Visualize Attention Maps

Attention-map export makes per-head weighting behavior observable and debuggable.

Final Model Comparison

A fair capstone comparison keeps prompt, seed, and evaluation protocol fixed across model variants.