SignBridge

A local speech-to-ASL accessibility tool. Converts live spoken English into animated ASL sign language displayed on a realistic 3D avatar.

Stack: Python FastAPI (port 8000) + React/Vite (port 5173) · AMD RX 7900 GRE (ROCm/gfx1100) · 100% local · zero cost

Prerequisites

Tool	Purpose	Install
ROCm	GPU runtime for AMD	`amdgpu-install` from repo.radeon.com
Ollama 0.18+	gemma3:12b inference	ollama.com
whisper.cpp	Transcription (HIP build)	see below
uv	Python env/package manager	astral.sh/uv
bun	JS runtime/package manager	bun.sh

Environment (one-time, requires reboot)

Add these lines to /etc/environment:

HSA_OVERRIDE_GFX_VERSION=11.0.0

This tells ROCm to treat the RX 7900 GRE (gfx1100) as a supported target. Without it both Ollama and whisper.cpp fall back to CPU.

whisper.cpp (HIP build)

whisper.cpp must be built with the ROCm HIP backend enabled. A build without -DGGML_HIP=ON will silently use CPU and miss the 700 ms latency target by an order of magnitude.

cd whisper.cpp
rm -rf build && mkdir build && cd build

# Configure with HIP — use hipcc as the compiler
HSA_OVERRIDE_GFX_VERSION=11.0.0 cmake .. \
  -DGGML_HIP=ON \
  -DCMAKE_C_COMPILER=/opt/rocm/bin/hipcc \
  -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \
  -DCMAKE_BUILD_TYPE=Release

cmake --build . --config Release -j$(nproc)

# Download the medium model (~1.5 GB)
cd ..
bash models/download-ggml-model.sh medium

Verify GPU use:

HSA_OVERRIDE_GFX_VERSION=11.0.0 ./build/bin/whisper-cli \
  -m models/ggml-medium.bin \
  -f samples/jfk.wav \
  --no-timestamps

Expected output includes whisper_backend_init_gpu: using device XXXXX (not no GPU found). Transcription must complete in under 1 second.

Ollama

# Pull the model once
ollama pull gemma3:12b

# Verify GPU execution — response should appear in under 5 seconds
ollama run gemma3:12b "Say hello in one word"

# Confirm GPU (not CPU) in the process list
ollama ps

Backend

Uses uv for fast, reproducible Python environments.

cd backend

# Create venv and install dependencies
uv venv
uv pip install -r requirements.txt

# Run (activate venv first, or prefix with uv run)
source .venv/bin/activate
uvicorn main:app --reload --port 8000

Health check:

curl localhost:8000/health
# Expected: {"whisper":"ok","ollama":"ok","latency_ms":...}

Frontend

Uses bun for fast installs and dev server.

cd frontend
bun install
bun run dev   # starts on :5173

Open http://localhost:5173 — both status indicators must show green.

Phase 1: Push-to-Talk Transcription

Phase 1 adds the full Spacebar push-to-talk path from browser microphone capture to backend transcription.

Flow

Hold Spacebar to start recording in the browser (MediaRecorder + getUserMedia).
Release Spacebar to stop recording and send audio to POST /api/transcribe.
Backend runs whisper.cpp and returns:

{ "text": "Good morning how are you", "latency_ms": 620.4 }

UI displays transcript text and latency bar values:

Transcription = whisper subprocess latency
Round-trip = client observed time from key release to response rendered

API Contract

Endpoint: POST /transcribe
Request: multipart/form-data, field name file
Success: { "text": string, "latency_ms": number }
Error: 422 with { "detail": "..." } for whisper execution errors

Quick Backend Validation

With backend running on :8000:

curl -X POST http://localhost:8000/transcribe -F "file=@whisper.cpp/samples/jfk.wav"

Expected: JSON with non-null latency_ms and a transcription string.

Phase 2: Text to ASL Gloss + OOV Handling

Phase 2 adds gloss translation and out-of-vocabulary (OOV) resolution between transcription and keypoint lookup.

Backend Additions

POST /gloss:
- Input: { "text": string }
- Output: { "tokens": GlossToken[], "latency_ms": number }
- Flow:
  1. Translate English text to ASL gloss with gemma3:12b
  2. Strip <think>...</think> from model output
  3. Annotate tokens against backend/data/wlasl_vocabulary.json
  4. If OOV exists, run constrained rewrite using top vocabulary list
POST /resolve_oov:
- Input: { "tokens": string[], "oov_tokens": string[] }
- Output: { "tokens": GlossToken[], "latency_ms": number }

GlossToken shape:

{
  "token": "WEATHER",
  "status": "in_library | rewritten | fingerspell",
  "original": "UNFORTUNATELY"
}

Frontend Additions

usePipeline.js now exposes both transcribe(audioBlob) and gloss(text)
New GlossPanel renders persistent chips:
- green: in_library
- yellow: rewritten (tooltip shows original token)
- red: fingerspell
App flow now chains gloss() immediately after transcript is set

Quick Phase 2 Validation

With backend running on :8000:

curl -s -X POST http://localhost:8000/gloss \
  -H "Content-Type: application/json" \
  -d '{"text": "I want to go to the store tomorrow"}' | python3 -m json.tool

Expected: uppercase token chips with statuses and endpoint latency_ms.

Phase 3: Sign Library + Keypoint Lookup

Phase 3 adds the offline WLASL keypoint pipeline and live POST /keypoints lookup endpoint.

Offline scripts project (`scripts/`)

Use a separate uv project so heavy extraction dependencies do not impact backend runtime:

cd scripts
uv sync

scripts/pyproject.toml dependencies:

mediapipe
opencv-python
numpy

1) Extract normalized keyframes from WLASL videos

cd scripts
uv run python extract_keypoints.py <path/to/WLASL_v0.3.json> <path/to/wlasl/videos> [output_json]

Default output is backend/data/wlasl_keypoints.json.

2) Ingest keyframes into SQLite

cd scripts
uv run python ingest_sqlite.py <path/to/wlasl_keypoints.json> [db_path]

Default DB output is backend/data/wlasl_keypoints.db.

3) Validate Phase 3 gate

sqlite3 backend/data/wlasl_keypoints.db "SELECT frame_count FROM signs WHERE gloss='HELLO';"

Expected: a number between 20 and 80.

Runtime endpoint

POST /keypoints
- Input: { "tokens": GlossToken[] }
- Output: { "sequences": KeypointSequence[], "latency_ms": number }
- fingerspell tokens are expanded into letter sequences with 4 neutral frames between letters.
Phase 4: 3D Avatar Rendering + IK

Phase 4 replaces the avatar placeholder with a live Three.js scene that loads a ReadyPlayerMe GLB and drives both arms with CCD IK.

Frontend dependencies
```
cd frontend
bun add three mathjs
```
Notes:
- Import addons from three/addons/... paths.
- mathjs is installed now for Phase 5 spline work.
Avatar asset
- Place the ReadyPlayerMe export at frontend/public/avatar.glb.
- Vite serves this at /avatar.glb.
Implemented files
- frontend/src/components/AvatarScene.jsx
  - Owns all Three.js setup and render loop.
  - Loads /avatar.glb with GLTFLoader.loadAsync.
  - Builds CCDIKSolver chains for left and right arms.
  - Uses ResizeObserver on the panel container.
  - Includes DEV_IK_TEST oval wrist target animation for gate validation.
- frontend/src/lib/boneMap.js
  - Exports MediaPipe to ReadyPlayerMe bone mapping for pose and hand joints.
- frontend/src/App.jsx
  - Replaces Phase 4 placeholder with <AvatarScene />.
Run
```
cd frontend
bun run dev
```
Gate checks
1. Gate 4.1: Confirm renderer.info.render.frame increments roughly 60 frames per second.
2. Gate 4.2: Avatar is visible with no console errors and required arm bone names are present.
3. Gate 4.3: Hardcoded right wrist IK target moves wrist to the fixed location.
4. Gate 4.4: DEV_IK_TEST oval motion moves the right arm smoothly without snapping.

Phase 5: Keypoints to Avatar Animation

Phase 5 adds the full keypoint-driven playback path: gloss tokens now fetch keyframes, keyframes are upsampled with a natural cubic spline, transitions are inserted between signs, and AvatarScene consumes queued frames each render tick.

Implemented files

frontend/src/lib/spline.js
- Natural cubic spline with mathjs lusolve().
- upsample(values, targetCount) for scalar tracks.
- upsampleFrame(frames, targetCount) for pose + hand landmarks.
- buildTransitionFrames(framesA, framesB, transitionCount) using 3 tail + 3 head source frames.
frontend/src/hooks/useAnimationQueue.js
- AnimationQueue with add(), tick(), clear(), and isPlaying.
- Module-level frozen NEUTRAL_FRAME.
- Transition insertion plus neutral buffer insertion for long wrist jumps.
frontend/src/components/AvatarScene.jsx
- Accepts queueRef prop.
- Computes shoulder midpoint once after avatar load.
- Applies wrist IK targets and finger rotations from queue frames.
- Uses renderer setAnimationLoop() and queue.tick(deltaMs) playback.
frontend/src/hooks/usePipeline.js
- Adds fetchKeypoints(tokens) via /api/keypoints.
frontend/src/App.jsx
- Creates and passes animation queue ref.
- Clears queue on new push-to-talk result.
- Chains gloss -> keypoints -> queue.add(...) for each resolved token.

Run

cd frontend
bun run dev

Phase 5 gate checks

Gate 5.1: HELLO sign plays and wrists track expected targets.
Gate 5.2: spline-upsampled playback is visibly smoother than raw frame stepping.
Gate 5.3: HELLO -> THANK-YOU plays without snapping at boundary.
Gate 5.4: 5 queued signs play continuously without timing gaps.

Phase Status

Phase	Description	Gate	Status
0.1	ROCm / GPU	`rocminfo \| grep gfx1100`	Pass
0.2	Ollama + gemma3:12b	`ollama ps` shows GPU	Pass
0.3	whisper.cpp HIP	transcription < 1s, GPU backend active	Rebuilding
0.4	FastAPI `/health`	`curl :8000/health` returns ok	Pass
0.5	React status panel	`:5173` shows green dots	Pass
2.1	Gloss translation	`/gloss` returns clean uppercase tokens	In Progress
2.2	OOV detection	OOV words are non-green statuses	In Progress
2.3	OOV rewrite	`/resolve_oov` rewrites to known tokens when possible	In Progress
2.4	Visual chips	Gloss chip colors and tooltip render in UI	In Progress
3.1	Sign library ingest	HELLO frame count is in 20-80	In Progress

Architecture

[Browser :5173]
  Spacebar held  →  MediaRecorder → WAV blob
  Spacebar up    →  POST /transcribe → English text
                 →  POST /gloss      → ASL gloss chips (green/yellow/red)
                 →  POST /keypoints  → keyframe arrays
                 →  cubic spline     → animation queue
  Three.js loop  →  CCDIKSolver drives avatar bones @ 60fps

[FastAPI :8000]
  /health      →  binary check + ollama list
  /transcribe  →  whisper.cpp subprocess → text + latency_ms
  /gloss       →  gemma3:12b via Ollama → gloss tokens (strip_thinking applied)
  /resolve_oov →  gemma3:12b semantic rewrite with vocab constraint
  /keypoints   →  SQLite lookup → keyframe arrays

Latency Targets

Layer	Target
Whisper (transcription)	≤ 700 ms
gemma3:12b (gloss)	≤ 800 ms
gemma3:12b (OOV rewrite)	≤ 600 ms
SQLite keypoint lookup	≤ 50 ms
Total to first sign	≤ 2.5 s

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
.cursor/plans		.cursor/plans
backend		backend
frontend		frontend
scripts		scripts
tests		tests
.gitignore		.gitignore
AGENTS.md		AGENTS.md
BUILDER-PROMPT.md		BUILDER-PROMPT.md
DEBUGGER-PROMPT.md		DEBUGGER-PROMPT.md
Phases.md		Phases.md
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

SignBridge

Prerequisites

Environment (one-time, requires reboot)

whisper.cpp (HIP build)

Ollama

Backend

Frontend

Phase 1: Push-to-Talk Transcription

Flow

API Contract

Quick Backend Validation

Phase 2: Text to ASL Gloss + OOV Handling

Backend Additions

Frontend Additions

Quick Phase 2 Validation

Phase 3: Sign Library + Keypoint Lookup

Offline scripts project (scripts/)

1) Extract normalized keyframes from WLASL videos

2) Ingest keyframes into SQLite

3) Validate Phase 3 gate

Runtime endpoint

Phase 4: 3D Avatar Rendering + IK

Frontend dependencies

Avatar asset

Implemented files

Run

Gate checks

Phase 5: Keypoints to Avatar Animation

Implemented files

Run

Phase 5 gate checks

Phase Status

Architecture

Latency Targets

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Offline scripts project (`scripts/`)

Packages