LoRA from scratch

A minimal, readable implementation of LoRA (Low-Rank Adaptation) in PyTorch, built to understand the method by implementing its core mechanics directly.

It includes a LoRALinear layer, a helper for injecting LoRA into existing nn.Linear modules, and a few experiment scripts covering the low-rank adapter, weight merging, and a small LLM fine-tuning run. The focus is on making the mechanics easy to read and inspect rather than building a full training library.

What is included

• lora/layers.py - LoRA wrapper around nn.Linear
• lora/inject.py - helper functions for replacing target linear layers with LoRA layers
• experiments/test_lora_linear.py - small sanity check for the LoRA layer
• experiments/test_merge.py - checks merge / unmerge behavior
• experiments/train_tiny_lora.py - toy rank experiment on a linear layer
• experiments/train_tiny_llm_lora.py - small LLM LoRA fine-tuning experiment

Setup

This project uses uv.

uv sync

Run the small experiments

uv run python experiments/test_lora_linear.py
uv run python experiments/test_merge.py
uv run python experiments/train_tiny_lora.py

The LLM experiment uses Qwen/Qwen2.5-0.5B-Instruct, so it will download the model from Hugging Face:

uv run python experiments/train_tiny_llm_lora.py