Skip to content

[BUG] Duplicate detection reads large files because threshold uses SizeUnit math #341

Description

@equationzhao

Describe the bug
Duplicate detection is intended to read only selected bytes for files larger than 16 KiB, but the threshold is defined as int64(16 * KiB). In this codebase KiB is a SizeUnit value based on internal display-unit math, not a byte count.

This makes the threshold much larger than 16 KiB, so fileHash can read whole large files with os.ReadFile, which is risky for memory usage and performance.

To Reproduce

  1. Inspect internal/content/duplicate.go:
    var thresholdFileSize = int64(16 * KiB)
  2. Compare with internal/content/size.go, where KiB is not the byte value 1024.
  3. Run duplicate detection over large files and observe that files below the inflated threshold take the whole-file path.

Screenshots
Not applicable.

Info (please complete the following information):

  • OS: all
  • Version: v0.31.2

Additional context
The threshold should likely be a byte count such as 16 * 1024 or use an explicit conversion helper that returns bytes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions