MedVault — Fine-tuned Qwen2.5-VL-7B for Handwritten Prescription Extraction

MedVault fine-tunes Qwen2.5-VL-7B-Instruct, a vision-language model, to read handwritten medical prescriptions and extract structured medicine information. Training uses 4-bit QLoRA for memory-efficient fine-tuning on consumer/free-tier GPUs, with experiment tracking via Weights & Biases and checkpoints pushed to the Hugging Face Hub.

Repository Structure

medvault_fine-tuned_VLM/
├── Notebook/
│   ├── medvault_finetuning.ipynb    # QLoRA fine-tuning pipeline
│   └── medvault_evaluation.ipynb    # Inference + evaluation on the test set
├── helper.py                        # Shared utility functions
├── pyproject.toml                   # Project dependencies
├── uv.lock                          # Locked dependency versions
├── .python-version
├── .gitignore
├── LICENSE
└── README.md

Dataset/, Dataset.zip, wandb/, medvaultlog/, .venv/, and .env are git-ignored — they hold the raw dataset, experiment logs, and local secrets, and are not tracked in this repo.

Installation

This project uses uv for dependency management.

git clone https://github.com/Zero-iinfinity/medvault_fine-tuned_VLM.git
cd medvault_fine-tuned_VLM
uv sync

You'll also need a Hugging Face token (HF_TOKEN) and, optionally, a Weights & Biases API key (WANDB_API_KEY) set as environment variables (or in a .env file) to pull/push models and log training runs.

Model & Dataset

	Link
Base model	Qwen/Qwen2.5-VL-7B-Instruct
Fine-tuned model	Zero-iinfinity/medvault-2026-07-10_14.54.26
Dataset	Zero-iinfinity/Qwen2.5-VL-7B-Instruct (dataset)
Method	QLoRA, 4-bit (NF4) quantization

Using the Model

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig
from qwen_vl_utils import process_vision_info

MODEL_PATH = "Zero-iinfinity/medvault-2026-07-10_14.54.26"

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    MODEL_PATH,
    quantization_config=quant_config,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model.eval()

def predict(image_path):
    messages = [
        {"role": "user", "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": "Extract medicine information."}
        ]}
    ]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to(model.device)

    with torch.no_grad():
        generated_ids = model.generate(**inputs, max_new_tokens=64)

    trimmed = [out[len(inp):] for inp, out in zip(inputs["input_ids"], generated_ids)]
    return processor.batch_decode(trimmed, skip_special_tokens=True)[0]

print(predict("path/to/prescription.png"))

See Notebook/medvault_evaluation.ipynb for the full evaluation pipeline.

Fine-tuning Details

Full pipeline in Notebook/medvault_finetuning.ipynb.

Hyperparameter	Value
Epochs	3
Batch size (per device)	1
Gradient accumulation steps	8
Max sequence length	256
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate	1e-4
LR scheduler	cosine
Optimizer	paged_adamw_32bit
Weight decay	0.001
Quantization	4-bit NF4 (QLoRA)
Precision	bf16 on Ampere+ GPUs, fp16 fallback otherwise

Evaluation Results

Evaluated on a held-out test set of handwritten prescription word images.

Overall accuracy: 0.90

Class	Precision	Recall	F1-score	Support
Ace	1.00	1.00	1.00	10
Aceta	1.00	1.00	1.00	10
Alatrol	0.90	0.90	0.90	10
Amodin	0.00	0.00	0.00	0
Amodis	1.00	0.80	0.89	10
Atrizin	1.00	0.80	0.89	10
Azithrocin	0.00	0.00	0.00	0
Platinum	0.00	0.00	0.00	0
Macro avg	0.61	0.56	0.58	50
Weighted avg	0.98	0.90	0.94	50

Amodin, Azithrocin, and Platinum have zero support in this test batch — they show up only as false-positive predictions, which is why their precision/recall/F1 are all 0 and macro avg is pulled down relative to the weighted avg.

License

Released under the MIT License.