HOPE Deduplication Engine

The HOPE Deduplication Engine (HDE) is a service in the HOPE ecosystem that detects duplicate individuals in a dataset by comparing their facial photographs. Client systems register batches of images through a REST API, trigger asynchronous processing (face quality assessment, encoding, and comparison), and read back duplicate findings with similarity scores.

Documentation

Full documentation is at https://unicef.github.io/hope-dedup-engine/:

• Concepts — architecture, set lifecycle, data model, face pipeline
• Integration Guide — for client systems using the API
• Administration — deployment, configuration, operations
• Development — contributing to the engine itself

Quick start (development)

Requirements: uv and Docker, or see the full setup guide.

git clone https://github.com/unicef/hope-dedup-engine.git
cd hope-dedup-engine
docker compose up --build

Then open http://localhost:8000/admin/ (adm@hde.org / 123) and the API docs at http://localhost:8000/api/rest/swagger/.

For a native (non-Docker) environment:

uv venv .venv
uv sync

./manage.py env --develop > .envrc  # create initial development configuration
direnv allow .                      # enable environment
createdb hope_dedup_engine          # create postgres database on localhost
./manage.py upgrade
./manage.py runserver