Shakespeare GPT
A decoder-only GPT built from scratch in PyTorch with every component implemented by hand; trained to generate Shakespeare.
Problem
Using nn.Transformer teaches you an API, not a transformer. The goal was the
internals: every component of a decoder-only GPT by hand, with only nn.Linear,
nn.Embedding, and nn.LayerNorm as primitives.
How it was solved
Six transformer blocks with hand-written multi-head causal self-attention, learned positional embeddings, pre-norm residuals, and dropout. Attention scores, causal masking, the multi-head reshape, and sampling are all written out.
Trained on ~1MB of tiny-shakespeare, on Apple Silicon’s MPS backend. Generation samples from the predicted distribution reshaped by temperature: 0.5 is grammatical but repetitive, 1.2 inventive but error-prone, 0.8 the sweet spot.
Results
Validation loss reached ~1.19. From the prompt "To be" at temperature 0.8:
To be patience than the less blood with our crown.
KING RICHARD III:
Then shall dam, of the stars what you sleep-change now,
Which you could be patience keep their worships,Real words, correct spelling, play structure, Shakespearean phrasing. A small character-level model captures style far more than meaning.