Skip to content

CLAUDE.md

Latest commit

 

History

History
180 lines (132 loc) · 5.55 KB

CLAUDE.md

File metadata and controls

180 lines (132 loc) · 5.55 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

ethnicolr2 is a PyTorch-based machine learning package that predicts race and ethnicity from names using LSTM neural networks. It's a modernized version of the original ethnicolr package, trained on US Census data and Florida voter registration data. The package supports prediction based on:

  • Last name only (census model or Florida model)
  • First and last name combined (Florida full name model)

Development Commands

Installation and Setup

# Install in development mode
uv sync

# Install with development dependencies
uv sync --group dev

# Install with testing dependencies
uv sync --group test

# Install with documentation dependencies
uv sync --group docs

# Install all optional dependencies
uv sync --all-groups

Testing

# Run all tests
uv run pytest

# Run tests with verbose output
uv run pytest --verbose

# Run specific test file
uv run pytest tests/test_census_lookup.py

# Run tests with coverage
uv run coverage run -m pytest && uv run coverage report

Code Quality

# Format and lint code with ruff
uv run ruff check --fix .
uv run ruff format .

Documentation

# Build documentation (from docs/ directory)
cd docs && uv run sphinx-build -b html source build

# View documentation
open docs/build/html/index.html

Code Architecture

Core Components

  1. Model Classes (ethnicolr_class.py):

    • EthnicolrModelClass: Base class providing common prediction functionality
    • Contains shared methods for data preprocessing, tensor conversion, and model inference
    • Handles batch processing with PyTorch DataLoader

  2. Neural Network Models (models.py):

    • LSTM: Custom LSTM implementation with embedding layer
    • Configured for character-level sequence processing
    • Outputs log-softmax probabilities for race/ethnicity categories

  3. Prediction Modules:

    • pred_fl_ln_lstm.py: Florida last name model (LastNameLstmModel)
    • pred_fl_fn_lstm.py: Florida full name model (FullNameLstmModel)
    • pred_cen_ln_lstm.py: Census last name model (CensusLstmModel)

  4. Data Handling:

    • dataset.py: EthniDataset class for PyTorch data loading
    • census_ln.py: Census data lookup functionality
    • utils.py: Command-line argument parsing utilities

Model Files and Data

  • Pre-trained Models (pinned on gojiberries/ethnicolr2 on Hugging Face):

    • lstm_lastname_gen.pt: Florida last name LSTM model
    • lstm_fullname.pt: Florida full name LSTM model
    • census_lstm_lastname.pt: Census last name LSTM model
    • Corresponding .joblib vectorizer files for each model

  • Training Data (in data/):

    • Census surname data from 2000 and 2010
    • Florida voter registration data samples
    • Example input files for testing

Prediction Categories

The models predict one of five race/ethnicity categories:

  • nh_white: Non-Hispanic White
  • nh_black: Non-Hispanic Black
  • hispanic: Hispanic
  • asian: Asian
  • other: Other

Model-Specific Parameters

  • Florida models: 30-character max name length, 256 hidden units
  • Census models: 15-character max name length, different category ordering
  • Full name models: 47-character max name length
  • All models use 2-layer LSTM with batch size 64

Key Dependencies

The authoritative dependency versions are declared in pyproject.toml and locked in uv.lock. PyTorch provides model inference, scikit-learn and joblib load the published vectorizers, pandas handles tabular data, and huggingface-hub resolves immutable model assets.

Testing Strategy

Tests are organized by functionality in the tests/ directory:

  • test_census_lookup.py: Census data lookup tests
  • test_models.py: Model loading and prediction tests
  • test_cli.py: Command-line interface tests
  • test_validation.py: Input validation tests
  • test_regression.py: Regression tests for bug fixes

All tests use pytest framework and include data validation, prediction accuracy checks, and error handling verification.

Command Line Interface

The package provides CLI scripts for each prediction model:

# Census last name prediction
census_ln input.csv -l last_name -o output.csv

# Florida last name prediction
pred_fl_last_name input.csv -l last_name -o output.csv

# Florida full name prediction (requires both first and last name)
pred_fl_full_name input.csv -l last_name -f first_name -o output.csv

# Census last name prediction with LSTM
pred_census_last_name input.csv -l last_name -o output.csv

# Download pre-trained models
ethnicolr2_download_models

Documentation

GitHub Pages Documentation

The project documentation is built with Sphinx and automatically deployed to GitHub Pages:

  • Live Documentation: https://appeler.github.io/ethnicolr2/
  • Auto-deployment: Triggered on pushes to main branch via .github/workflows/docs.yml
  • Configuration: docs/source/conf.py dynamically reads metadata from pyproject.toml
  • Theme: Furo theme (furo) - modern, clean documentation theme

Building Documentation Locally

# Install documentation dependencies
uv sync --group docs

# Build documentation
cd docs && uv run sphinx-build -b html source build

# View locally
open docs/build/html/index.html

Documentation Features

  • Dynamic version: Automatically reads from pyproject.toml
  • Dynamic authors: Extracted from project metadata
  • Auto-generated API docs: Uses Sphinx autodoc
  • GitHub Pages integration: Automatic deployment with .nojekyll file