Day 96: Building nanoGPT (part 1): the architecture in code
Assembling a GPT from the pieces
Karpathy's 'Let's build GPT' walks you through nanoGPT — a complete, minimal GPT. Today you assemble the architecture: token + positional embeddings, a stack of transformer blocks (your Day-95 block), a final layer norm, and a linear head projecting to vocabulary logits. Because you understand every component, this is composition, and it demystifies GPT permanently.
import torch.nn as nn
class GPT(nn.Module):
def __init__(self, vocab_size, n_embd, n_head, n_layer, block_size):
super().__init__()
self.tok_emb = nn.Embedding(vocab_size, n_embd) # Day 92
self.pos_emb = nn.Embedding(block_size, n_embd) # Day 93
self.blocks = nn.Sequential(*[
TransformerBlock(n_embd, n_head) for _ in range(n_layer) # Day 95
])
self.ln_f = nn.LayerNorm(n_embd)
self.head = nn.Linear(n_embd, vocab_size) # logits over vocab
def forward(self, idx):
pos = torch.arange(idx.size(1), device=idx.device)
x = self.tok_emb(idx) + self.pos_emb(pos) # embed + position
x = self.ln_f(self.blocks(x))
return self.head(x) # next-token logitsCausal masking
A GPT predicts the *next* token, so during training each position may only attend to earlier positions — never peek at the future. This causal mask sets attention scores to future tokens to −∞ before softmax, zeroing their weight. It's the one addition that turns the bidirectional attention of Day 94 into the autoregressive, left-to-right attention a generative model needs.
This is genuinely how ChatGPT-class models are built
nanoGPT is small, but architecturally identical to production LLMs — the same embeddings, blocks, attention, and causal masking, just fewer layers and parameters. Scale changes capability, not structure. After building this, 'large language model' stops being a mystery box and becomes 'a big version of the thing I built'.
Key terms
- nanoGPT
- Karpathy's minimal, complete GPT implementation — architecturally identical to production LLMs at small scale.
- Causal mask
- Masking future positions in attention so each token attends only to itself and earlier tokens — required for autoregressive generation.
- Autoregressive
- Generating a sequence one token at a time, each conditioned on all previous tokens.
What does the causal mask enforce in a GPT during training?
