Transformer from Scratch

A small, readable PyTorch implementation for tracing the core ideas in Attention Is All You Need and applying them to decoder-only language modeling.

Status: Implementation scaffold
Focus: tensor shapes, causal attention, and autoregressive generation
Framework: PyTorch
Hardware budget: 1× RTX 3070

Summary

This repository implements token and positional embeddings, multi-head causal self-attention, feed-forward layers, residual connections, layer normalization, language-model loss, and autoregressive generation. The code favors explicit tensor transformations over framework shortcuts so every important shape can be inspected.

Reproduction target

Item	Paper / reference	This implementation
Attention	Scaled dot-product	Explicit QKᵀ / √d
Architecture	Encoder–decoder	Small decoder-only model
Objective	Sequence modeling	Next-token prediction
Dataset	[state selected corpus]	[record exact split]
Parameters	[reference]	[record actual count]
Validation loss	[reference]	[record verified result]

The adaptation from the original encoder–decoder architecture must remain explicit in the README and article.

Implementation scope

• token and learned positional embeddings
• query, key, and value projections
• multi-head reshape and merge operations
• causal attention mask
• residual paths and layer normalization
• feed-forward network
• cross-entropy objective and sampling

Paper-to-Code map

Paper item	Code
Scaled dot-product attention	src/minigpt/model.py::CausalSelfAttention
Multi-head split / merge	src/minigpt/model.py::CausalSelfAttention.forward
Position information	src/minigpt/model.py::MiniGPT.forward
Feed-forward network	src/minigpt/model.py::FeedForward
Residual and normalization	src/minigpt/model.py::Block

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest

mkdir -p data
cp /path/to/your/text.txt data/input.txt
python scripts/train.py --input data/input.txt

Shape trace

Record one concrete batch here and keep it synchronized with the code.

Tensor	Shape	Concrete example
Token IDs	(B, T)
Embedded input	(B, T, C)
Q, K, V	(B, H, T, D)
Attention scores	(B, H, T, T)
Logits	(B, T, V)

Additional experiment

Compare pre-norm and post-norm blocks with the same tokenizer, model size, seeds, optimizer, update count, and evaluation procedure.

Norm style	Validation loss	Gradient behavior	Notes
Pre-norm
Post-norm

What I learned

• [Why the causal mask is applied before softmax]
• [Where head dimensions are split and merged]
• [How normalization placement changes optimization]

Links

• Technical article: [URL]
• Portfolio: [URL]