Skip to content

Can we remove the local DB and operate directly on the CSV + local assets? #145

Description

@jonborchardt

Summary
Much of the slowness and instability appears tied to the local DB layer. Requesting an option (or migration) to run without the DB: read/write directly to the CSV and local asset files.

Context / Problem

  • Current local DB introduces performance overhead and occasional failures.
  • For my workflows, the CSV already represents the source of truth for records; assets live on disk.
  • The DB adds complexity (schema/versioning, sync, caching) that doesn’t seem to pay off for small–medium projects.

Proposal

  • Provide a “DB-less mode” that:
    • Reads records directly from the CSV.
    • Writes changes atomically back to the CSV (append+rename pattern or temp file with checksum).
    • Resolves assets by path on disk (no DB indirection).
    • Uses in-memory indices per session only (rebuild quickly at startup).
  • Keep existing DB mode as optional for very large datasets, if needed.

Questions

  • Are there specific features that hard-require the DB (e.g., complex queries, transactions across multiple tables, offline diff/merge)?
  • What’s the minimum needed indexing to keep UI snappy without persisting a DB?
  • Can we gate this behind a project setting: “Storage backend: CSV (direct) | Local DB”?

Acceptance Criteria

  • Opening, editing, searching, and exporting a project with N≈50k rows works without the DB enabled.
  • Cold start index build on 50k rows completes within a reasonable baseline.
  • All CRUD flows read/write only the CSV and file system assets.
  • No data loss in the face of app crash during write (verified with fault injection).
  • A project-level toggle (and migration path) is documented and tested.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions