filesieve

filesieve is a command-line utility for finding exact duplicate files and moving duplicate copies into an alternate directory while leaving one canonical original in place.

It is optimized for large media collections with:

• staged exact hashing (size filter -> quick hash -> full hash -> byte verify),
• optional perceptual media similarity clustering (images + video),
• persistent SQLite signatures cache for faster repeated runs.

Project overview

• Walks one or more base directories recursively.
• Moves only exact byte-identical duplicates.
• Keeps canonical file by oldest mtime_ns, then lexicographic path.
• Emits perceptual media clusters as report-only output (no auto-move).

See Duplicate detection algorithm for details.

Supported Python versions

filesieve currently supports Python 3.10+.

Installation (with uv)

Install from local source

uv sync

Install as a tool (console entry point)

uv tool install .

After install, the filesieve command is available in your shell.

CLI usage

General form:

filesieve [OPTIONS] BASE_DIR [BASE_DIR ...]

Core options

• -a, --alternate DUP_DIR: move exact duplicates here.
• -c, --config FILE: optional config path.
• --mode {exact,media}: duplicate mode (media default).
• --cache PATH: SQLite cache path override.
• --no-cache: disable persistent cache.
• --hash-workers N: worker threads for exact hashing.
• --media-workers N: worker threads for perceptual media stage.
• --ffmpeg PATH: explicit ffmpeg path or executable name.
• --ffprobe PATH: explicit ffprobe path or executable name.
• --report-similar PATH: write perceptual media clusters JSON.
• --move-similar: move perceptual duplicates to the alternate directory, keeping the highest-resolution file.

Examples

Exact duplicate cleanup only:

filesieve --mode exact --alternate /tmp/sieve/dups ~/Videos

Media mode with report output:

filesieve --mode media --report-similar ./similar.json --alternate /tmp/sieve/dups ~/Photos ~/Videos

--report-similar requires --mode media; exact mode only considers byte-identical files.

Move lower-resolution perceptual duplicates after reviewing a report:

filesieve --mode media --move-similar --report-similar ./similar.json \
  --alternate /tmp/sieve/dups ~/Photos

Perceptual moves are opt-in because similarity matching can produce false positives. Within each cluster, FileSieve keeps the file with the greatest pixel area. Resolution ties prefer the larger file, then the oldest modification time, then the lexicographically first path.

Run through uv:

uv run filesieve --alternate /tmp/sieve/dups ./library

For full details:

filesieve --help

Media organizer mode (Plex preset)

filesieve also supports organizing video media into a Plex-friendly structure with idempotent state tracking, duplicate routing, dry-run previews, and optional native UI.

CLI dry-run example:

filesieve --organize-media --organize-target /media/library --organize-report ./organize-report.json ~/Downloads ~/Staging

Apply changes:

filesieve --organize-media --organize-apply --organize-target /media/library ~/Downloads ~/Staging

Open native UI (source selection, target configuration, progress, pause/continue/stop):

filesieve --organize-ui --organize-target /media/library

Organizer options:

• --organize-config PATH: YAML organizer config (see config/organize.yaml).
• --organize-state-db PATH: SQLite state for idempotency and re-runs.
• --organize-apply: apply moves (default is dry-run).
• --organize-report PATH: write JSON report of planned/executed operations.

Behavior notes:

• Non-media files are ignored.
• Unknown media naming falls into Unsorted.
• Duplicates are moved to Duplicates and canonical picks highest parsed quality.
• On destination conflicts, version suffixes are appended.
• On Windows, cross-drive moves use copy+verify+delete for safety.

Configuration

Pass a config with --config /path/to/sieve.conf.

Precedence order:

1. CLI args
2. config file values
3. in-code defaults

Example config:

[global]
dup_dir:/tmp/sieve/dups
mode:media
cache_db:.filesieve-cache.sqlite
hash_workers:8
media_workers:2

[media]
enabled:true
image_hamming_threshold:8
video_hamming_threshold:32
video_frame_hamming_threshold:12
duration_bucket_seconds:2

Safety model

• File moves happen only for exact duplicates after byte-for-byte verification.
• Perceptual matches are advisory unless --move-similar is explicitly used.
• If FFmpeg tools are unavailable, perceptual stage is skipped automatically.

Additional documentation

• Duplicate detection algorithm and caveats
• Development workflow (setup, lint, test, build)
• Release workflow (versioning + publishing)
• Migration from legacy script usage