Skip to content

ROADMAP.adoc

Latest commit

 

History

History
197 lines (147 loc) · 7.33 KB

ROADMAP.adoc

File metadata and controls

197 lines (147 loc) · 7.33 KB
Raw

Evidence Graph (bofig) - Roadmap

Table of Contents

1. Overview

18-month development plan for the Evidence Graph, an infrastructure platform for pragmatic epistemology in investigative journalism.

2. Phase 1: Proof of Concept (Months 1-6)

2.1. v0.1.0 - Foundation [COMPLETE]

Released: 2025-11-22

  • Elixir/Phoenix application with ArangoDB multi-model database

  • GraphQL API (Absinthe): 15 queries, 11 mutations

  • PROMPT epistemological scoring (6 dimensions)

  • Claims, Evidence, Relationships, Navigation Paths data models

  • Graph traversal algorithms (evidence chains, shortest path, contradiction detection)

  • UK Inflation 2023 test dataset (7 claims, 30 evidence items)

  • D3.js force-directed graph visualisation

  • RSR compliance: LICENSE, SECURITY.md, CONTRIBUTING, CHANGELOG

2.2. v1.0.0 - Phase 1 PoC [RELEASED]

Released: 2026-02-21

  • Phoenix 1.8 LiveView frontend (5 interactive pages)

  • User authentication (phx.gen.auth with bcrypt)

  • Zotero REST API integration (import, export, batch-import, sync-status)

  • D3.js force-directed graph + radar chart visualisations

  • Audience-weighted navigation paths (6 audience types)

  • Production deployment config (Containerfile, nginx, systemd)

  • Podman containerisation with podman-compose

  • NUJ user testing protocols (task script, consent, feedback, decision matrix)

  • A2ML v2.1 Cyberwar-Ready Trustfile

  • 257 tests, 0 failures, 0 compile warnings

  • Full RSR compliance with all contractiles

2.3. v1.1.0 - Zotero Browser Extension (Month 3-4)

  • Zotero browser extension for one-click evidence import

  • Two-way sync between Zotero library and Evidence Graph

  • Batch operations for large reference collections

  • PROMPT score suggestions from metadata analysis

2.4. Decision Point: Month 3

Criteria: NUJ journalist user testing (target: 25 participants)

  • Continue: If journalists find the tool useful for real investigations

  • Pivot: If core assumptions about evidence navigation are wrong

  • Metrics: Task completion rate, time-on-task, System Usability Scale (SUS)

3. Phase 2: Platform + Lithoglyph Integration (Months 7-12)

3.1. v1.2.0 - Lithoglyph as Evidence Store [IN PROGRESS]

Started: 2026-03-13

The critical architectural shift: Lithoglyph becomes the primary evidence store, replacing ArangoDB for domain data. ArangoDB is retained only for graph edges (relationships) until Lithoglyph’s Factor GQL runtime supports graph traversals.

  • Lithoglyph HTTP client (Req) for evidence CRUD — DONE

  • Batch importer GenServer with progress tracking via PubSub — DONE

  • NER entity extraction wired into import pipeline — DONE

  • Entity resolution (exact + fuzzy Jaro-Winkler + auto-create) — DONE

  • Evidence→entity :mentions relationship edges — DONE

  • Migrate evidence reads to Lithoglyph GQL queries (stop reading from ArangoDB evidence collection)

  • Migrate evidence writes to Lithoglyph (insert via GQL-DT with provenance, not ArangoDB)

  • Migrate entity reads/writes to Lithoglyph (entities collection)

  • Migrate claims to Lithoglyph (claims collection with PROMPT as first-class types)

  • Keep ArangoDB for relationships edge collection only (graph traversals)

  • PROMPT scores stored as GQL-DT BoundedNat 0 100 types (compile-time verified)

3.2. v2.0.0 - Multi-Investigation Platform

  • Multi-investigation dashboard with cross-referencing

  • Real-time collaborative editing via Phoenix PubSub

  • Advanced D3.js visualisations (timeline, heatmap, Sankey diagrams)

  • Full-text search across evidence corpus (via Lithoglyph Factor GQL)

  • Role-based access control (admin, journalist, reviewer, reader)

  • API rate limiting and key management

  • IPFS provenance integration for tamper-evident storage

  • Deployment to Hetzner Cloud (EU data sovereignty)

  • Lithoglyph managed instance (replaces ArangoDB Oasis)

3.3. Decision Point: Month 9

Criteria: Platform stability and user retention

  • Scale: If 50+ active users and stable infrastructure

  • Consolidate: Focus on reliability before new features

4. Phase 3: Lithoglyph-Native + Production (Months 13-18)

4.1. v3.0.0 - Full Lithoglyph Migration

The long-term architecture: bofig is a pure Lithoglyph client. ArangoDB is fully removed. All data lives in Lithoglyph with GQL/GQL-DT queries.

4.1.1. 3A: Graph Edges → Lithoglyph (v3.0.0-alpha)

  • Migrate relationships edge collection from ArangoDB to Lithoglyph

  • Implement graph traversals (evidence chains, shortest path, contradiction detection) in Factor GQL

  • Requires: Lithoglyph Factor GQL runtime gains graph query support (TRAVERSE, SHORTEST_PATH)

  • ArangoDB fully removed — no more multi-database architecture

  • PostgreSQL retained for user auth only (phx.gen.auth)

4.1.2. 3B: GQL-DT Query Integration (v3.0.0-beta)

  • Replace raw GQL queries with GQL-DT (dependently typed) queries

  • PROMPT scores verified at compile-time: BoundedNat 0 100 instead of runtime range checks

  • Evidence insertions require Rationale (NonEmptyString) — enforced by Lean 4 type checker

  • Proof certificates for evidence integrity (SHA-256 hash verification via GQL-DT)

  • Type-safe evidence chain traversals with proof obligations

4.1.3. 3C: Production Launch (v3.0.0)

  • Public API with documentation and SDK (GQL endpoint, not REST)

  • Formal security audit and penetration testing

  • Lean 4 formal proofs for PROMPT score calculation correctness

  • Integration with academic citation databases (via Lithoglyph provenance)

  • Cross-investigation evidence linking (Lithoglyph multi-tenancy)

  • EU funding application

  • Academic paper on boundary objects in journalism tooling

  • NUJ partnership formalisation

4.2. Decision Point: Month 12

Criteria: Sustainability and community

  • Open: If community contributors emerge, open governance model

  • Sustain: If grant funding secured, dedicated development

  • Archive: If insufficient adoption, document lessons learned

5. Database Migration Path

Phase Evidence/Claims/Entities Relationships (Graph) Auth

Phase 1 (v1.0.0)

ArangoDB

ArangoDB (edge collection)

PostgreSQL

Phase 2 (v1.2.0–v2.0.0)

Lithoglyph (primary), ArangoDB (read fallback during migration)

ArangoDB (edge collection)

PostgreSQL

Phase 3 (v3.0.0)

Lithoglyph (sole store)

Lithoglyph (Factor GQL graph queries)

PostgreSQL

The migration is incremental: each collection moves independently. Evidence moves first (highest value), then entities, then claims, then graph edges last (requires Lithoglyph graph query support).

6. Success Metrics

Phase Metric Target

Phase 1

NUJ participants complete test tasks

25 participants, >70% task completion

Phase 2

Active monthly users

50+ journalists/researchers

Phase 3

Investigations using the platform

10+ real investigations

7. Technology Stack

  • Backend: Elixir 1.18+ / Phoenix 1.8+ / OTP 27

  • Evidence Store: Lithoglyph (GQL/GQL-DT, narrative-first, audit-grade)

  • Graph DB: ArangoDB 3.11+ (relationships only, Phase 2; removed in Phase 3)

  • Auth DB: PostgreSQL 16 (user accounts only)

  • Query Language: GQL (user tier) / GQL-DT (admin tier, Lean 4 verified)

  • API: GraphQL (Absinthe) + REST (Zotero)

  • Frontend: Phoenix LiveView + D3.js

  • Container: Podman + podman-compose

  • Hosting: Hetzner Cloud (EU)

  • CI/CD: GitHub Actions (19 workflows)