Repository navigation
Expand file tree
/
Copy patharchitecture.html
More file actions
2196 lines (2015 loc) · 132 KB
/
Copy patharchitecture.html
File metadata and controls
2196 lines (2015 loc) · 132 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Continuum — Architecture & Mental Model</title>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<style>
:root {
--bg: #0f1117;
--surface: #161b22;
--surface2: #21262d;
--border: #30363d;
--text: #e6edf3;
--text-muted: #8b949e;
--text-faint: #484f58;
--accent: #58a6ff;
--accent-dim: #1f6feb;
--green: #3fb950;
--yellow: #d29922;
--orange: #f0883e;
--red: #f85149;
--purple: #bc8cff;
--pink: #ff7b72;
--sidebar-w: 256px;
--header-h: 56px;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
background: var(--bg);
color: var(--text);
font-family: 'Inter', system-ui, sans-serif;
font-size: 15px;
line-height: 1.7;
display: flex;
min-height: 100vh;
}
/* ── Sidebar ── */
.sidebar {
position: fixed;
top: 0; left: 0; bottom: 0;
width: var(--sidebar-w);
background: var(--surface);
border-right: 1px solid var(--border);
overflow-y: auto;
padding: 24px 0 40px;
z-index: 100;
}
.sidebar-logo {
padding: 0 20px 20px;
border-bottom: 1px solid var(--border);
margin-bottom: 16px;
}
.sidebar-logo .product {
font-size: 18px;
font-weight: 700;
color: var(--text);
letter-spacing: -0.3px;
}
.sidebar-logo .tagline {
font-size: 11px;
color: var(--text-muted);
margin-top: 2px;
}
.sidebar-section {
padding: 8px 20px 4px;
font-size: 10px;
font-weight: 700;
letter-spacing: 0.08em;
text-transform: uppercase;
color: var(--text-faint);
}
.sidebar a {
display: block;
padding: 5px 20px;
color: var(--text-muted);
text-decoration: none;
font-size: 13px;
border-left: 2px solid transparent;
transition: all 0.12s;
}
.sidebar a:hover { color: var(--text); background: var(--surface2); }
.sidebar a.active { color: var(--accent); border-left-color: var(--accent); background: rgba(88,166,255,0.06); }
.sidebar-doc-link {
margin: 16px 20px 0;
padding: 10px 14px;
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 8px;
font-size: 12px;
color: var(--text-muted);
}
.sidebar-doc-link a { padding: 0; display: inline; color: var(--accent); }
/* ── Main ── */
main {
margin-left: var(--sidebar-w);
max-width: 860px;
padding: 48px 56px 96px;
width: 100%;
}
/* ── Typography ── */
h1 { font-size: 32px; font-weight: 700; letter-spacing: -0.5px; color: var(--text); margin-bottom: 8px; }
h2 { font-size: 22px; font-weight: 700; letter-spacing: -0.3px; color: var(--text); margin: 48px 0 16px; padding-top: 48px; border-top: 1px solid var(--border); }
h3 { font-size: 16px; font-weight: 600; color: var(--text); margin: 28px 0 10px; }
h4 { font-size: 13px; font-weight: 600; color: var(--text-muted); text-transform: uppercase; letter-spacing: 0.06em; margin: 20px 0 8px; }
p { color: var(--text-muted); margin-bottom: 14px; }
strong { color: var(--text); font-weight: 600; }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
.lead {
font-size: 16px;
color: var(--text-muted);
line-height: 1.8;
margin-bottom: 32px;
padding-bottom: 32px;
border-bottom: 1px solid var(--border);
}
/* ── Callout boxes ── */
.callout {
border-radius: 8px;
padding: 14px 18px;
margin: 16px 0;
border-left: 3px solid;
font-size: 14px;
}
.callout.info { background: rgba(88,166,255,0.06); border-color: var(--accent); color: var(--text-muted); }
.callout.warn { background: rgba(210,153,34,0.08); border-color: var(--yellow); color: var(--text-muted); }
.callout.tip { background: rgba(63,185,80,0.06); border-color: var(--green); color: var(--text-muted); }
.callout strong { color: var(--text); }
/* ── Code ── */
code {
font-family: 'JetBrains Mono', monospace;
font-size: 13px;
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 4px;
padding: 1px 6px;
color: var(--orange);
}
pre {
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 8px;
padding: 20px;
overflow-x: auto;
margin: 16px 0;
}
pre code {
background: none;
border: none;
padding: 0;
font-size: 13px;
color: var(--text);
line-height: 1.6;
}
.comment { color: var(--text-faint); }
.kw { color: var(--pink); }
.str { color: var(--green); }
.fn { color: var(--accent); }
.num { color: var(--purple); }
/* ── Diagrams ── */
.diagram-wrap {
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 10px;
padding: 24px;
margin: 20px 0;
overflow-x: auto;
}
.mermaid {
display: flex;
justify-content: center;
}
/* ── Tables ── */
table {
width: 100%;
border-collapse: collapse;
margin: 16px 0;
font-size: 13px;
}
th {
background: var(--surface2);
color: var(--text-muted);
font-weight: 600;
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.06em;
padding: 10px 14px;
border: 1px solid var(--border);
text-align: left;
}
td {
padding: 9px 14px;
border: 1px solid var(--border);
color: var(--text-muted);
vertical-align: top;
}
td code { font-size: 12px; }
td strong { color: var(--text); }
tr:hover td { background: rgba(255,255,255,0.02); }
/* ── Tags / Badges ── */
.badge {
display: inline-block;
padding: 2px 8px;
border-radius: 20px;
font-size: 11px;
font-weight: 600;
letter-spacing: 0.03em;
}
.badge.green { background: rgba(63,185,80,0.15); color: var(--green); }
.badge.blue { background: rgba(88,166,255,0.15); color: var(--accent); }
.badge.orange { background: rgba(240,136,62,0.15); color: var(--orange); }
.badge.purple { background: rgba(188,140,255,0.15);color: var(--purple); }
.badge.red { background: rgba(248,81,73,0.15); color: var(--red); }
.badge.yellow { background: rgba(210,153,34,0.15); color: var(--yellow); }
/* ── Grid ── */
.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 16px 0; }
.grid-3 { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 12px; margin: 16px 0; }
.card {
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 8px;
padding: 16px;
}
.card h4 { margin-top: 0; margin-bottom: 6px; color: var(--accent); font-size: 12px; }
.card p { margin-bottom: 0; font-size: 13px; }
/* ── Step list ── */
.steps { counter-reset: step; list-style: none; }
.steps li {
counter-increment: step;
position: relative;
padding: 12px 0 12px 44px;
border-bottom: 1px solid var(--border);
font-size: 14px;
color: var(--text-muted);
}
.steps li:last-child { border-bottom: none; }
.steps li::before {
content: counter(step);
position: absolute;
left: 0;
top: 12px;
width: 28px; height: 28px;
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 50%;
display: flex;
align-items: center;
justify-content: center;
font-size: 12px;
font-weight: 700;
color: var(--accent);
}
.steps li strong { color: var(--text); }
/* ── Section anchor spacing ── */
section { scroll-margin-top: 24px; }
h2 { scroll-margin-top: 24px; }
/* ── Horizontal rule ── */
hr { border: none; border-top: 1px solid var(--border); margin: 32px 0; }
/* ── File tree ── */
.file-tree {
font-family: 'JetBrains Mono', monospace;
font-size: 13px;
background: var(--surface2);
border: 1px solid var(--border);
border-radius: 8px;
padding: 20px;
line-height: 1.9;
color: var(--text-muted);
white-space: pre;
overflow-x: auto;
}
.file-tree .dir { color: var(--accent); font-weight: 600; }
.file-tree .key { color: var(--orange); }
.file-tree .cmt { color: var(--text-faint); font-style: italic; }
</style>
</head>
<body>
<!-- ═══════════════════════════════════════════════════════
SIDEBAR
══════════════════════════════════════════════════════════ -->
<nav class="sidebar">
<div class="sidebar-logo">
<div class="product">Continuum</div>
<div class="tagline">Novel Continuity Engine — Architecture Guide</div>
</div>
<div class="sidebar-section">Overview</div>
<a href="#what-is-this">What Is This?</a>
<a href="#mental-model">Mental Model</a>
<a href="#system-map">System Map</a>
<div class="sidebar-section">Core Pipeline</div>
<a href="#pipeline-overview">Pipeline Overview</a>
<a href="#continuity-policy">Blocking vs Advisory</a>
<a href="#extraction-passes">13 Extraction Passes</a>
<a href="#entity-resolution">Entity Resolution</a>
<a href="#chapter-cap">Chapter Cap</a>
<div class="sidebar-section">Data Layer</div>
<a href="#db-design">Database Design</a>
<a href="#core-tables">Core Tables</a>
<a href="#sota-tables">SOTA Tables</a>
<a href="#temporal-model">Temporal Model</a>
<div class="sidebar-section">API & Frontend</div>
<a href="#api-layer">API Layer</a>
<a href="#frontend-pages">Frontend Pages</a>
<div class="sidebar-section">Key Concepts</div>
<a href="#canon-facts">Canon Facts</a>
<a href="#commitments">Commitments</a>
<a href="#knows-edges">Theory of Mind</a>
<a href="#entity-graph">Entity Graph</a>
<a href="#custom-entity-types">Custom Entity Types</a>
<div class="sidebar-section">Advanced Modules</div>
<a href="#retrieval">Hybrid Retrieval</a>
<a href="#critic">Continuity Critic</a>
<a href="#state-materializer">State Materializer</a>
<a href="#eval-harness">Eval Harness</a>
<div class="sidebar-section">Codebase</div>
<a href="#file-map">Key File Map</a>
<a href="#current-branch">History: the SOTA & Redesign Branches</a>
<div class="sidebar-doc-link">
← <a href="index.html">Start Here</a><br>
<a href="reference.html">API & Schema Reference →</a><br>
<a href="state-of-the-system.html">State of the System →</a><br>
<a href="blog/README.md">Blog: Building Continuum →</a>
</div>
</nav>
<!-- ═══════════════════════════════════════════════════════
MAIN CONTENT
══════════════════════════════════════════════════════════ -->
<main>
<!-- HERO -->
<section id="what-is-this">
<h1>Continuum</h1>
<p><strong>October 18 review:</strong> chapter history uses <code>story_record_versions</code>, <code>story_record_history</code>, and <code>story_records_at</code>. The snapshot fields are retained after a <a href="snapshot-design.html">measured comparison with status-only storage</a>. Naming and migration decisions are recorded in <a href="../naman-review.md">Naman’s review resolution</a>.</p>
<p class="lead">
A <strong>novel continuity engine</strong> that reads raw chapter text, extracts structured knowledge
via LLM, and stores it in PostgreSQL so authors and readers can query anything —
"where was Asha in chapter 3?", "what commitments are still unresolved?",
"who knows the secret?". The result is a living wiki that grows chapter by chapter.
</p>
<div class="grid-3">
<div class="card">
<h4>Ingest</h4>
<p>Raw chapter text → chunked, sent to LLM with story context</p>
</div>
<div class="card">
<h4>Extract</h4>
<p>13-pass LLM extraction of entities, events, threads, flags, scenes, dynamics, knowledge, commitments, canon facts</p>
</div>
<div class="card">
<h4>Query</h4>
<p>25+ wiki-style views served by FastAPI, rendered in React — including entity graph, scenes, canon facts, and custom entity types</p>
</div>
</div>
<div class="callout info">
<strong>Status: redesign complete (2026-07-13/14); write refusal added 2026-08-24,
unified into a single caller-agnostic critique 2026-09-04.</strong>
Continuum is a <strong>continuity analyzer</strong>, not a chapter generator — that product
decision is final, not provisional. Two spines run over one Postgres database: the
<strong>write spine</strong> (<code>analyze_chapter</code>) turns raw chapter text — from the
Process page/CLI (<code>source='human'</code>) or an MCP <code>save_chapter</code> call from a
writing agent (<code>source='agent'</code>) — into structured knowledge, and the
<strong>read layer</strong> (<code>backend/reads/</code>) serves every wiki page, API route,
and MCP tool from that same data, cutoff-aware by construction. The write spine does not branch
on who is writing: every caller runs CRITIQUE → INGEST → EXTRACT → PERSIST → MATERIALIZE →
RECORD, with the critique run once, up front. Only the <em>consequence</em> of a FAIL varies,
via the caller's <code>on_continuity_fail</code> policy — see
<a href="#pipeline-overview">Pipeline Overview</a> and
<a href="#continuity-policy">Blocking vs Advisory</a> below. See <a href="state-of-the-system.html">State of the System</a> for the
full history of how it got here.
</div>
</section>
<!-- MENTAL MODEL -->
<section id="mental-model">
<h2>Mental Model</h2>
<p>
Think of Continuum as a <strong>ETL pipeline for fiction</strong>. It does for novels what a data warehouse
does for business events — continuously ingests raw narrative text and produces a structured,
queryable knowledge graph.
</p>
<div class="grid-2">
<div class="card">
<h4>The Pipeline (Write Path)</h4>
<p>
Chapter text → LLM extracts facts → deduplication → entity resolution → PostgreSQL.
Everything flows in one direction. Replacement restores the earlier metadata and rebuilds the affected chapters; explicit identity repairs are retroactive.
</p>
</div>
<div class="card">
<h4>The API (Read Path)</h4>
<p>
React frontend asks FastAPI for wiki views. All queries are "as-of chapter N" (the chapter cap),
using chapter state projections and changed metadata versions. Older imports have an explicit history baseline.
</p>
</div>
</div>
<h3>Three Layers of Abstraction</h3>
<table>
<thead>
<tr><th>Layer</th><th>What lives here</th><th>Key constraint</th></tr>
</thead>
<tbody>
<tr>
<td><strong>Entities</strong></td>
<td>Characters, locations, factions, objects, and user-defined custom types — the nouns of the story</td>
<td>Every entity gets a UUID in the <code>entities</code> table first</td>
</tr>
<tr>
<td><strong>Events</strong></td>
<td>Timestamped story events — the verbs. Now enriched with SVO triples and story-time ordering.</td>
<td>Always tied to a chapter; can reference any entity type via UUID arrays</td>
</tr>
<tr>
<td><strong>State</strong></td>
<td>Character state snapshots, scenes, relationships, knowledge edges, bitemporal possession/location edges</td>
<td>Every state row has a chapter reference → enables chapter-cap filtering</td>
</tr>
</tbody>
</table>
<div class="callout tip">
<strong>Key insight:</strong> There are no "current state" columns anywhere. State is always derived by querying up to a chapter cap. This is the design that enables spoiler-free browsing.
</div>
</section>
<!-- SYSTEM MAP -->
<section id="system-map">
<h2>System Map</h2>
<p>How the major modules connect:</p>
<div class="diagram-wrap">
<div class="mermaid">
graph LR
A["📄 Chapter Text\n(raw file / stdin)"] --> B["ingest_chapter\n(ingestion/)"]
B --> C[(chapters\ntable)]
C --> D["load_story_context\n(pipeline.py)"]
D --> E["sliding_window_chunks\n(chunker.py)"]
E --> F["ChapterExtractor\n(extractor.py)"]
F -->|13 passes| G["LLM\n(LiteLLM)"]
G --> F
F --> H["IntraExtractionDeduplicator\n(canonicalizer.py)"]
H --> H2["EntityCanonicalizer\n(canonicalizer.py)"]
H2 --> I["EntityResolver\n(resolver.py)"]
I --> J["persist_extraction\n(extraction/persist.py)"]
J --> K[(PostgreSQL\nknowledge graph)]
K --> L["FastAPI\n(api/)"]
L --> M["React\n(frontend/)"]
style A fill:#21262d,stroke:#30363d,color:#e6edf3
style K fill:#1f6feb,stroke:#58a6ff,color:#e6edf3
style G fill:#3d2b1f,stroke:#f0883e,color:#e6edf3
</div>
</div>
<h3>Directory Structure</h3>
<div class="file-tree">
<span class="dir">continuum/</span>
├── <span class="dir">backend/</span>
│ ├── <span class="dir">api/</span>
│ │ ├── <span class="key">app.py</span> <span class="cmt"># FastAPI app, mounts 19 routers</span>
│ │ ├── <span class="key">admin.py</span> <span class="cmt"># Novel create/delete + canon fact mutations (SQL-only, real Postgres)</span>
│ │ ├── <span class="key">schemas.py</span> <span class="cmt"># Pydantic response models (35+ types)</span>
│ │ ├── <span class="key">jobs.py</span> <span class="cmt"># In-memory job status store</span>
│ │ └── routes/ <span class="cmt"># One file per entity/feature domain — no SQL, calls reads/ (enforced by reads/tests/test_contract.py)</span>
│ │ ├── entity_graph.py <span class="cmt"># Unified entity + rich edge graph</span>
│ │ └── entity_types.py <span class="cmt"># Custom entity type CRUD + genre presets</span>
│ ├── <span class="dir">reads/</span> <span class="cmt"># Read layer: SQL-only, cutoff-aware, no FakeDB branches. The sole presentation reader — api/queries.py is gone.</span>
│ │ ├── <span class="key">db.py</span> <span class="cmt"># get_db() — the read layer's DB provider</span>
│ │ ├── <span class="key">common.py</span> <span class="cmt"># resolve_cutoff(), max_chapter_for(), merge_story_edges()</span>
│ │ ├── <span class="key">novels.py</span> <span class="cmt"># list_novels, get_novel</span>
│ │ ├── <span class="key">chapters.py</span> <span class="cmt"># list_chapters (+ critique summary), list_scenes</span>
│ │ ├── <span class="key">characters.py</span> <span class="cmt"># list_characters, get_character_detail, get_character_page (MCP + wiki share this)</span>
│ │ ├── <span class="key">continuity.py</span> <span class="cmt"># list_flags, get_chapter_critique, list_critiques</span>
│ │ ├── <span class="key">world.py</span> <span class="cmt"># locations, objects, factions, entity types, custom entities</span>
│ │ ├── <span class="key">timeline.py</span> <span class="cmt"># list_timeline — backs the wiki timeline route and the MCP timeline_events tool</span>
│ │ ├── <span class="key">graphs.py</span> <span class="cmt"># relationship_graph, entity_graph, list_shared_dynamics</span>
│ │ ├── <span class="key">relationship_types.py</span> <span class="cmt"># resolve_symmetric() — sole source of relationship symmetry</span>
│ │ ├── <span class="key">threads.py</span> <span class="cmt"># list_threads (+ status_at_cutoff)</span>
│ │ ├── <span class="key">commitments.py</span> <span class="cmt"># list_commitments (+ status_at_cutoff)</span>
│ │ ├── <span class="key">knowledge.py</span> <span class="cmt"># knows_edges, located_in_edges, possesses_edges, canon_facts</span>
│ │ └── search.py <span class="cmt"># hybrid search — backs /api/search and the MCP search_story tool</span>
│ ├── <span class="dir">pipeline/</span>
│ │ ├── <span class="key">pipeline.py</span> <span class="cmt"># Master orchestration — start here</span>
│ │ ├── <span class="key">config.py</span> <span class="cmt"># All tuneable settings (frozen dataclass)</span>
│ │ ├── <span class="dir">db/</span>
│ │ │ ├── <span class="key">schema.sql</span> <span class="cmt"># Source of truth for DB structure</span>
│ │ │ ├── client.py <span class="cmt"># psycopg3 connection pool wrapper</span>
│ │ │ ├── entity_merge.py <span class="cmt"># merge_entities() — repoint every reference, absorb aliases, drop the source (cross-type merge = reclassification)</span>
│ │ │ └── duplicates.py <span class="cmt"># find_duplicate_candidates() — duplicate-looking pairs, split same-type vs cross-type</span>
│ │ ├── <span class="dir">extraction/</span>
│ │ │ ├── <span class="key">extractor.py</span> <span class="cmt"># ChapterExtractor — drives 13 LLM passes</span>
│ │ │ ├── <span class="key">prompts.py</span> <span class="cmt"># All prompt templates + JSON schemas</span>
│ │ │ ├── <span class="key">presets.py</span> <span class="cmt"># Genre presets for custom entity types</span>
│ │ │ ├── canonicalizer.py <span class="cmt"># Intra-extraction dedup + cross-chapter canonicalizer</span>
│ │ │ ├── resolver.py <span class="cmt"># Entity name → DB ID resolution (exact/alias/partial)</span>
│ │ │ ├── chunker.py <span class="cmt"># Sliding window tokenizer</span>
│ │ │ ├── persist_deltas.py <span class="cmt"># Persist typed state_deltas (reference-only resolution)</span>
│ │ │ ├── persist_canon.py <span class="cmt"># Persist canon_facts (lock/contradiction semantics)</span>
│ │ │ └── persist_extras.py <span class="cmt"># Persist SOTA tables (scenes, knows, etc.)</span>
│ │ ├── <span class="dir">critic/</span> <span class="cmt"># ContinuityCritic: 4 typed checks</span>
│ │ │ ├── runner.py <span class="cmt"># Orchestrates all checks → CritiqueReport</span>
│ │ │ ├── adapter.py <span class="cmt"># Draft/extraction → critic bridge (extract_draft_claims, build_draft_chapter, build_draft_from_extraction)</span>
│ │ │ ├── types.py <span class="cmt"># DraftChapter, CritiqueReport, Finding</span>
│ │ │ └── checks/ <span class="cmt"># entity, knowledge, possession, commitments</span>
│ │ ├── <span class="dir">retrieval/</span> <span class="cmt"># HybridRetriever: BM25 + dense + RRF + MMR</span>
│ │ │ ├── hybrid.py <span class="cmt"># Orchestrator: BM25 → dense → RRF → MMR</span>
│ │ │ ├── bm25.py <span class="cmt"># Full-text search (tsvector)</span>
│ │ │ ├── dense.py <span class="cmt"># Vector ANN search (HNSW)</span>
│ │ │ ├── fusion.py <span class="cmt"># Reciprocal rank fusion</span>
│ │ │ ├── mmr.py <span class="cmt"># Max marginal relevance diversification</span>
│ │ │ └── types.py <span class="cmt"># RetrievalQuery, RetrievalResult</span>
│ │ ├── <span class="dir">state/</span> <span class="cmt"># StateMaterializer: derives projections from state_deltas</span>
│ │ │ ├── materializer.py <span class="cmt"># Sole writer of character_states; also writes located_in_edges, possesses_edges</span>
│ │ │ ├── replay.py <span class="cmt"># Folds state_deltas and knows_edges into snapshots</span>
│ │ │ └── types.py <span class="cmt"># StateSnapshot, LocationFact, PossessionFact</span>
│ │ ├── embeddings.py <span class="cmt"># LiteLLM embedding; explicit mock hash vectors</span>
│ │ ├── llm.py <span class="cmt"># Shared litellm loader + lenient JSON parser</span>
│ │ ├── entity_tables.py <span class="cmt"># Single source of the entity type → table map</span>
│ │ └── ingestion/ingest.py <span class="cmt"># Raw chapter INSERT</span>
│ ├── <span class="dir">mcp_server/</span> <span class="cmt"># FastMCP stdio server (novel-mcp): 13 tools for writing agents</span>
│ │ ├── server.py <span class="cmt"># @mcp.tool definitions + entry point (main); read tools call reads/ directly, no SQL</span>
│ │ └── queries.py <span class="cmt"># Writer-only glue: check_continuity, save_chapter (all read tools moved to reads/ in the read-layer migration)</span>
│ ├── <span class="dir">evals/</span> <span class="cmt"># Offline eval harness: golden fixture novel + answer key, scoring lib, retrieval/critic/extraction-fidelity/entity-resolution evals — see Eval Harness below</span>
│ └── pyproject.toml <span class="cmt"># Dependencies + script entry points</span>
└── <span class="dir">frontend/src/</span>
├── <span class="key">App.tsx</span> <span class="cmt"># Router + QueryClient setup, 24 routes</span>
├── <span class="key">api.ts</span> <span class="cmt"># Type definitions + all fetch functions</span>
├── components/
│ ├── <span class="key">Sidebar.tsx</span> <span class="cmt"># Navigation (including dynamic custom-type links) + chapter cap slider</span>
│ └── Layout.tsx
└── routes/ <span class="cmt"># 24 page components</span>
</div>
</section>
<!-- PIPELINE -->
<section id="pipeline-overview">
<h2>Pipeline Overview</h2>
<p>
The pipeline lives in <code>backend/pipeline/pipeline.py</code>. The entry point for processing a
chapter is <code>analyze_chapter()</code>. The
<a href="architecture-conformance.md">2026-09-19 conformance audit</a> separates the
implemented boundaries from open consistency gaps. Here's the processing flow:
</p>
<ol class="steps">
<li>
<strong>0 · CRITIQUE — once when enabled</strong> — runs before extraction. Advisory callers may skip it; blocking callers always require it. <code>critique_draft()</code> (<code>pipeline/critic/service.py</code>) runs <code>extract_draft_claims()</code> (one LLM call) → <code>build_draft_chapter()</code> → <code>ContinuityCritic.critique()</code> against the raw, unextracted text. It returns a <code>DraftCritique</code> and never raises: <code>status="ok"</code> carries a real verdict, <code>"error"</code> means the critique itself blew up, and <code>"unavailable"</code> means it never ran (<code>CRITIC_ENABLED=false</code>, or no real model — an empty mock draft would pass every check vacuously, so mock mode is an outage rather than a pass). This is the <em>only</em> critique the chapter gets; phase 5 persists this report rather than computing a second one. Running before chunking and the 13-pass extractor is what makes a refusal cost one LLM call instead of thirteen — and, more importantly, what keeps refused text out of the write tier entirely, since <code>delete_chapter_data</code> cannot undo the entity rows, <code>plot_threads</code> upserts and <code>canon_facts</code> updates a rejected draft would leave behind. What a FAIL costs is the caller's <code>on_continuity_fail</code> policy, not this function's business — see <a href="#continuity-policy">Blocking vs Advisory</a> below.</li>
<li>
<strong>Load story context</strong> — <code>load_story_context()</code> fetches existing characters, locations, open plot threads, recent events, and custom entity types from the DB. Characters and locations are <em>selected, not dumped</em>: entities whose name or alias appears in the chapter text always make the cut, the remainder back-fills by recency, and the lists are capped at <code>CONTEXT_MAX_CHARACTERS</code> (default 40) / <code>CONTEXT_MAX_LOCATIONS</code> (default 30) so prompt size stays bounded as the cast grows. The result is injected into every LLM prompt so the model knows what already exists.
</li>
<li>
<strong>Chunk the text</strong> — <code>sliding_window_chunks()</code> splits the chapter into 2,000-token windows with 200-token overlap using tiktoken. The encoding is derived from <code>DEFAULT_MODEL</code> (<code>o200k_base</code> for GPT-4o-class models, and as the estimator for any model tiktoken cannot map — every Gemini and Claude model). There is no whitespace fallback: a silently coarser window changes both chunk size and paragraph structure with nothing to detect it afterward, so a tokenizer that will not load raises. Each chunk is processed independently.
</li>
<li>
<strong>LLM extraction (13 passes per chunk)</strong> — <code>ChapterExtractor</code> runs 13 sequential LLM calls per chunk. See <a href="#extraction-passes">Extraction Passes</a> for details.
Chat and embedding calls have a 60-second attempt timeout and up to three retries
for transient provider errors. Retries honor HTTP <code>Retry-After</code> and
Gemini's JSON <code>RetryInfo</code>, capped at 60 seconds per wait. An exhausted
extraction request or empty/invalid JSON aborts processing before persistence;
it is never converted to an empty extraction that can silently erase data.
</li>
<li>
<strong>Merge across chunks</strong> — <code>merge_extractions()</code> deduplicates the results from all chunks by name/description (naïve string matching at this stage; objects are keyed by name + owner so same-named items with different owners survive the merge).
</li>
<li>
<strong>Intra-extraction deduplication</strong> — <code>IntraExtractionDeduplicator</code> makes one focused LLM call per entity type (including the novel's custom types) to group name variants ("Jane", "Jane B", "Miss Bennet") and rewrites supported references to an unambiguous established name or a validated model-nominated input name — across new entities, deltas, events (characters, locations, objects, <em>and factions</em>), relationships, dynamics, scenes, learnings, and custom entities. Group members are validated against the actually-extracted names, and output targets must be real group members or saved canonical names of the single matched established identity. Renamed lists are also de-duplicated in place.
</li>
<li>
<strong>Cross-chapter canonicalization</strong> — <code>EntityCanonicalizer</code> matches names not yet in the DB against the per-type roster. Names whose normalized form (<code>normalize_name()</code>: case, whitespace, surrounding punctuation, leading article) equals exactly one roster name/alias merge <em>deterministically with zero LLM cost</em>; only the rest go to one LLM call per type, with the roster capped to the <code>CANONICALIZER_MAX_ROSTER</code> (default 80) most string-similar entries so prompt size stays bounded as the novel grows. Matches confirmed by strong evidence (a verbatim anchor, or unambiguous lexical closeness for non-characters) are appended to the entity's <code>aliases[]</code> — custom entity types store theirs on <code>entities.aliases</code>; reasoning-only matches are instead applied as a rename to the current extraction (no permanent alias), so the merge still takes effect this chapter.
</li>
<li>
<strong>Entity resolution</strong> — <code>EntityResolver</code> maps every entity name to an existing DB entity (exact name → alias → character partial-name match → create new). Returns UUIDs. Only the <code>new_entities</code> pass is allowed to <em>create</em> characters; every other pass resolves with <code>create=False</code> (resolve-or-skip) so a name it merely references can't mint a phantom character.
</li>
<li>
<strong>Begin chapter transaction and ingest raw text</strong> — after extraction and
canonicalization, <code>ingest_chapter()</code> inserts the chapter inside the same session
as its assertions. Canonicalizer alias writes currently occur before this session and
survive a later chapter failure; this is an open transaction-boundary gap.
</li>
<li>
<strong>Persist to DB</strong> — <code>extraction/persist.py::persist_extraction()</code> writes entities, events, relationships, shared dynamics, plot threads, continuity flags. It never writes <code>character_states</code>: state changes go through the typed <code>state_deltas</code> log instead, and <code>character_states</code> is the sole responsibility of the <a href="#state-materializer">State Materializer</a>, which runs at the end of <code>analyze_chapter</code>.
</li>
<li>
<strong>Persist state deltas</strong> — <code>persist_state_deltas()</code> (<code>persist_deltas.py</code>) writes the <code>state_deltas</code> extraction pass into the <code>state_deltas</code> table: reference-only resolution (<code>create=False</code>) for the subject character, and for possession/location kinds the object/location too, so a delta naming an unknown entity is dropped rather than minting one. Rows carry <code>subject_id</code>/<code>object_id</code> as universal <code>entities.id</code> and <code>location_id</code> as the typed <code>locations.id</code>, with an <code>ordinal</code> preserving narrative order within the chapter — this is the tier-2 log the state materializer replays.
</li>
<li>
<strong>Embed</strong> — <code>embed_chapter_and_events()</code> generates vector embeddings for each event description, plus the chapter summary only when no medium summary exists (otherwise <code>persist_multi_summaries()</code> embeds the chapter from <code>summary_medium</code>, so the extra call is skipped).
</li>
<li>
<strong>Persist SOTA extras</strong> — <code>persist_multi_summaries()</code>, <code>persist_scenes()</code>, <code>persist_knows_edges()</code>, <code>persist_commitments()</code>, <code>persist_canon_facts()</code> fill the newer tables added in the SOTA branch. Canon facts upsert unless locked — a locked fact contradicted by extraction becomes a <code>continuity_flag</code> (<code>flag_type='canon_contradiction'</code>) rather than an overwrite. Database and embedding failures abort the chapter transaction. Unresolved references and unmatched payoffs produce enrichment warnings returned to the caller.
</li>
<li>
<strong>4 · MATERIALIZE</strong> — for an append, runs after persistence commits; during replacement it joins the enclosing transaction: <code>StateMaterializer</code> replays <code>state_deltas</code> and <code>knows_edges</code> through the novel's latest chapter. A failure here is caught and logged, not raised — the already-committed chapter stays saved, and the result dict's <code>"materialized"</code> flag tells the caller a manual re-run (<code>python -m pipeline.state.cli</code>) is needed.
</li>
<li>
<strong>5 · RECORD — persist the phase-0 verdict</strong> — no second judgement. <code>persist_critique()</code> (<code>pipeline/critic/persist.py</code>) writes the report phase 0 already produced to <code>critique_reports</code>/<code>critique_findings</code>, replacing any prior report for the chapter. The only reason this is a separate phase is that <code>critique_reports.chapter_id</code> is a foreign key: the chapter has to exist before its verdict can be filed against it. Skipped when phase 0 was skipped (<code>CRITIC_ENABLED=false</code> or <code>run_critic: false</code>) or produced no report (an outage) — a missing verdict must not overwrite an existing one with an empty one. Like phase 4 this is derived and idempotent: a failure is caught and logged rather than raised, and <code>python -m pipeline.critic.cli --novel-id <uuid> [--chapter N]</code> re-judges one chapter or a whole back catalogue from stored text without re-extracting — the recovery path after a transient failure or a change to a check. Because the checks compare this chapter's claims against the world state <em>entering</em> it, judging pre-ingest is not just cheaper but semantically cleaner: the chapter's own canon facts are not yet committed, so a mention cannot be compared against the row it just produced. (The entity-mention check still bounds itself to <code>source_chapter < chapter_number</code>, which is what makes a standalone re-run correct too.)
</li>
</ol>
<div class="callout info" id="continuity-policy">
<strong>Blocking vs advisory (write refusal added 2026-08-24; unified 2026-09-04).</strong>
Originally this was a separate "agent write gate": a second module that critiqued the draft,
decided, and refused — while phase 5 critiqued the same prose a second time to produce the
report. Two invocations, two LLM calls, one set of checks. They are now one phase.
<strong>The judgement is caller-agnostic and runs once; only the consequence of a FAIL is a
policy.</strong>
<ul>
<li><strong><code>on_continuity_fail="warn"</code> — the default.</strong> Record the
findings and ingest anyway. Used by the Process page and the CLI, where a human is already
the review step: telling them to approve their own draft would be ceremony, not safety.
The report lands in <code>critique_reports</code> and surfaces in the Continuity view.</li>
<li><strong><code>on_continuity_fail="block"</code>.</strong> Refuse the write, park the
draft, return <code>{"ingested": false, "status", "submission_id", "reason", "fails",
"warns"}</code> — no exception, no <code>chapters</code> row. Asked for by exactly one
caller: <code>mcp_server/queries.py::save_chapter</code>, because an agent writes to canon
unattended and at volume. Being the MCP tool is what makes a caller an agent; it is not a
string any caller can pass, which is the failure mode the old <code>source != 'human'</code>
test had.</li>
</ul>
<strong>The invariant is a property of the policy, not of the row.</strong> A chapter written
through <code>save_chapter</code> either passed continuity or was human-overridden through the
Review queue with the blocking findings recorded on it. Rows from warning callers carry their
findings without that guarantee. The refused text is never lost — it lands in
<code>draft_submissions</code>, a table that is deliberately <em>not</em> canon and is never
read by any query serving the wiki, the API, or MCP tools other than the review queue itself.
<ul>
<li><strong>Fail-closed, not fail-open.</strong> A blocking caller refuses on every outcome
that is not a clean pass. If the critique throws — bad prompt, malformed
<code>CritiqueReport</code>, anything — <code>critique_draft()</code> catches it and
returns <code>status="error"</code>; the draft is parked (<code>reason: "error"</code>,
<code>status: "pending_review"</code>). If it never ran at all
(<code>status="unavailable"</code>: <code>CRITIC_ENABLED=false</code>, or mock mode)
the write is refused with nothing parked — there is no verdict for a human to
adjudicate, so <code>status</code> is <code>"refused"</code> and
<code>submission_id</code> is <code>null</code>. The caller holds the text and retries.
<strong>A missing verdict is never a pass</strong>, and
<code>run_critic=false</code> cannot disable the check a blocking caller depends on.</li>
<li><strong>Resubmission always clears the stale row.</strong> A pending row for a chapter is
superseded (marked <code>rejected</code>) atomically with parking a new failure or
committing a passing chapter. A failed extraction leaves the old draft pending. A passing resubmission does
not insert a row of its own; it only clears the one left by the earlier failure. Without
this, a chapter that failed once and then passed would leave a stale <code>pending</code>
row in the queue forever, pointing at superseded text a reviewer can no longer accept (the
chapter it names already exists).</li>
<li><strong>The human is the only override.</strong> A parked draft sits in the wiki's
Review Queue (<code>/novels/{id}/review</code>, <code>routes/Review.tsx</code>) until a
human accepts or rejects it. Rejecting just marks the row; the agent must revise and
resubmit. Accepting re-runs <code>analyze_chapter</code> under
<code>on_continuity_fail="warn"</code> — the reviewer <em>is</em> the review step — and,
critically, does <em>not</em> discard the findings that blocked the draft: they are written
through to <code>continuity_flags</code> as <code>accepted_override:<check></code>
rows, so a human-blessed contradiction stays visible in the Continuity view instead of
being silently absolved. If the human edited the text before accepting, the flag is
labelled <code>[accepted after edit; original draft FAILed]</code> rather than
<code>[accepted despite continuity FAIL]</code> — the re-ingest critiques the edited text
into <code>critique_reports</code>, but never re-adjudicates the blocking verdict, so the
flag records only that the <em>original</em> draft failed. The accepted chapter,
override findings, and conditional review-status update now share one transaction;
a failed update or competing rejection rolls back the chapter too.</li>
<li><strong>Human writes are advisory, not unchecked.</strong> The Process page and the CLI
run the identical four checks on the identical claims and get the identical findings;
they simply do not treat a FAIL as a veto. This is the part the old design got wrong by
omission: because the gate was a separate module reachable only via
<code>source != 'human'</code>, "human" was a string any caller could pass to skip the
check entirely. Now <code>source</code> is provenance only — it lands in
<code>chapters.source</code> and decides nothing.</li>
</ul>
See <a href="reference.html#schema-drafts"><code>draft_submissions</code> schema</a> and
<a href="reference.html#api-drafts">the review-queue API</a> in the Reference for the exact
shapes.
</div>
<div class="callout warn">
<strong>Relationship reads mean "in force at the cutoff" (fixed 2026-08-17).</strong>
Two defects made relationship answers wrong rather than merely incomplete.
<code>reads/characters.py</code> required <code>entity_type = 'character'</code> on
<em>both</em> endpoints, so every character↔object/location/faction edge was invisible from
the character's side — <code>GET /objects/{id}</code> showed "Aelric wields Duskbrand" while
<code>GET /characters/{aelric}</code> and the MCP <code>get_character</code> tool showed
nothing, and <code>rel_to_row</code>'s <code>other_entity_type</code> resolution was
unreachable code. Separately, <code>relationships.to_chapter</code> appeared in <em>zero</em>
WHERE clauses repo-wide, so every read meant "started before N", never "still true at N" — an
alliance that broke in chapter 6 was still reported as current at chapter 20. All five
relationship reads now filter <code>to_chapter</code> and <code>superseded_by_id</code>, and
<code>list_location_edges</code>/<code>list_possession_edges</code> filter
<code>superseded_by_id</code> under <code>active_only</code> so a character who moves twice in
one chapter no longer reads as being in two places at once.
</div>
<div class="callout warn">
<strong>A relationship persists until the text breaks it (fixed 2026-09-04).</strong>
<code>to_chapter</code> means the relationship <em>stopped being true</em>. The extraction
prompt never said so — the schema offered <code>from_chapter</code> and <code>to_chapter</code>
as bare <code>integer|null</code> fields with no definition — so the model filled
<code>to_chapter</code> with the chapter it was reading, meaning "this is what I saw here".
Every relationship was therefore stored as having ended the moment it began.
<strong>Absence of a re-mention was being recorded as an ending.</strong>
<br><br>
The read layer then did exactly what it was told. Because
<code>relationship_graph</code> derives its nodes from its edges, a character whose only
relationships were closed-on-arrival disappeared from the graph one chapter later. On a
Pride and Prejudice ingest the graph went from 10 nodes at chapter 1 to 8 at chapter 2, losing
a married couple the book never separates — a cutoff that <em>removed</em> people as it moved
forward. The prompt now defines the interval, and
<code>reads/tests/test_timeline_graphs.py</code> pins both halves of the contract: raising the
cutoff can never drop a node, and a relationship the text genuinely ends does leave the graph
at its <code>to_chapter</code>.
</div>
<div class="callout warn">
<strong>Entity identity is additive-only (fixed 2026-08-17).</strong> Two resolver defects
could silently and permanently corrupt identity, which for a memory layer is worse than a
missing answer — a wrong merge is unrecoverable without manual repair and poisons every later
chapter. The partial-name branch used <code>LIMIT 1</code> with no <code>ORDER BY</code>, so a
bare "Elizabeth" bound to whichever of Elizabeth Bennet / Elizabeth Elliot the planner
happened to return, <em>and then persisted that guess as an alias</em>. It now returns every
candidate and refuses to bind when more than one matches. Alias writes are also gated on
<code>create=True</code>: reference-only passes (scene POV, knows-edges, event actors, state
deltas) resolve for lookup and must not mutate identity. Conversely,
<code>_merge_aliases()</code> now folds extractor-supplied aliases into entities that already
exist — previously they were written only on create, so a nickname established in a later
chapter was discarded and its next bare use minted a duplicate character.
</div>
<div class="callout info">
<strong>Consistency boundaries (2026-09-19).</strong> A per-novel database lock
covers the entire workflow. Aliases and changed metadata versions share the
persistence transaction. <code>knows_edges</code> is the sole knowledge source.
A replacement restores the preceding metadata and rebuilds all later chapters
in one transaction, rolling back completely on failure. Unresolved enrichment
references are reported; database errors are never swallowed. See the
<a href="architecture-conformance.md">conformance report</a> for upgrade limits.
</div>
<div class="callout warn">
<strong>Embedding failures now raise instead of substituting a hash vector.</strong>
<code>EmbeddingService.embed_text()</code> used to catch every failure — rate limit, timeout,
missing key, wrong <code>dimensions</code> — and return <code>_hash_embedding()</code>, whose
SHA-256 output carries no semantic signal whatsoever: two strings differing only by a trailing
period measure ≈ −0.016 cosine against each other, while two entirely unrelated texts measure
≈ −0.001. Those vectors were written to <code>chapters</code>, <code>events</code>,
<code>scenes</code> and <code>commitments</code> and indexed by HNSW, permanently and
undetectably degrading dense retrieval for whatever they touched. Outside explicit mock mode
the service now raises <code>EmbeddingError</code>, and asserts the returned vector width
matches <code>EMBEDDING_DIMENSIONS</code> before handing it back.
<br><br>
Those four tables also carry an <code>embedding_model</code> column recording what produced
each vector (<code>mock:hash</code> for mock runs, otherwise the configured model). This is
not only about the historical rows: <code>USE_MOCK_LLM</code> writes hash vectors
<em>legitimately</em>, so ingesting with mock and later switching to a real model is the
normal development path and produces a silently mixed store. To find rows that no longer match
the current configuration:
<pre><code>SELECT embedding_model, count(*) FROM chapters GROUP BY 1;</code></pre>
Rows with <code>NULL</code> predate the column; rows tagged <code>mock:hash</code> carry no
semantic content and should be re-ingested before dense retrieval is trusted.
</div>
<div class="callout warn">
<strong>Note on idempotency:</strong> Re-processing a chapter requires passing <code>replace=True</code> (API: <code>"replace": true</code> in the <code>ProcessRequest</code> body). Without it, ingest rejects duplicate chapter numbers. With <code>replace=True</code>, <code>delete_chapter_data()</code> removes the following chapter-scoped derived rows before re-processing begins:
<ul style="margin: 8px 0 0 0; padding-left: 20px; color: var(--text-muted);">
<li><code>events</code>, <code>scenes</code>, <code>character_states</code>, <code>continuity_flags</code>, <code>shared_dynamics</code> — deleted by <code>chapter_id</code></li>
<li><code>relationships</code> rows that carry a <code>chapter_id</code> (i.e. those written during this chapter's ingest)</li>
<li><code>knows_edges</code> and <code>commitments</code> keyed by chapter number — deleted by <code>chapter_number</code></li>
</ul>
Entity rows introduced by the chapter (<code>characters</code>, <code>locations</code>, <code>factions</code>, <code>objects</code>, <code>entities</code>) and plot-thread upserts are <strong>not</strong> deleted — they survive and are resolved against on re-processing, so entity identity remains stable across replace runs.
</div>
</section>
<!-- EXTRACTION PASSES -->
<section id="extraction-passes">
<h2>13 Extraction Passes</h2>
<p>
Each pass is a separate LLM call. The schema for what each pass returns is in
<code>prompts.py → PASS_SCHEMAS</code>. The order matters — later passes can reference entities
found in earlier ones via the merged context.
</p>
<table>
<thead>
<tr>
<th>#</th>
<th>Pass Name</th>
<th>What It Extracts</th>
<th>DB Target</th>
</tr>
</thead>
<tbody>
<tr>
<td><span class="badge blue">1</span></td>
<td><code>chapter_summary</code></td>
<td>One-sentence chapter summary</td>
<td><code>chapters.summary</code></td>
</tr>
<tr>
<td><span class="badge blue">2</span></td>
<td><code>new_entities</code></td>
<td>New characters, locations (with <code>parent_location</code>), factions, objects (with <code>owner_name</code> + <code>distinguishing_properties</code>), and <strong>custom entities</strong> (genre-specific types defined per novel)</td>
<td><code>characters</code>, <code>locations</code>, <code>factions</code>, <code>objects</code>, <code>entities</code> (for custom types)</td>
</tr>
<tr>
<td><span class="badge blue">3</span></td>
<td><code>state_deltas</code></td>
<td>Typed state changes: possession gain/loss, location moves, knowledge learned, status updates (emotional_state/goals/physical_state/appearance/notes) — each with a verbatim evidence quote</td>
<td><code>state_deltas</code> (folded into <code>character_states</code> and the bitemporal edges by the materializer)</td>
</tr>
<tr>
<td><span class="badge blue">4</span></td>
<td><code>events</code></td>
<td>Narrative events with type, impact level, involved entities</td>
<td><code>events</code></td>
</tr>
<tr>
<td><span class="badge blue">5</span></td>
<td><code>thread_updates</code></td>
<td>Plot thread open/progress/close with linked events</td>
<td><code>plot_threads</code>, <code>thread_events</code></td>
</tr>
<tr>
<td><span class="badge blue">6</span></td>
<td><code>continuity_flags</code></td>
<td>Foreshadowing, setups, callbacks, planted details</td>
<td><code>continuity_flags</code></td>
</tr>
<tr>
<td><span class="badge blue">7</span></td>
<td><code>relationship_updates</code></td>
<td>Entity-to-entity relationship arcs with type, direction, chapter range</td>
<td><code>relationships</code></td>
</tr>
<tr>
<td><span class="badge blue">8</span></td>
<td><code>dynamics_updates</code></td>
<td>Chapter-level snapshots of how two entities interact (tone, tension). Multiple updates for the same pair in one chapter (in either order) are merged into a single row — descriptions concatenated — to satisfy the one-row-per-pair-per-chapter unique constraint</td>
<td><code>shared_dynamics</code></td>
</tr>
<tr>
<td><span class="badge blue">9</span></td>
<td><code>scene_segmentation</code></td>
<td>Scene boundaries with POV character, location, time anchor, present characters</td>
<td><code>scenes</code></td>
</tr>
<tr>
<td><span class="badge blue">10</span></td>
<td><code>multi_granularity_summaries</code></td>
<td>Summary at three granularities: short (tweet), medium (paragraph), long (recap)</td>
<td><code>chapters.summary_short</code>, <code>summary_medium</code>, <code>summary_long</code></td>
</tr>
<tr>
<td><span class="badge blue">11</span></td>
<td><code>knowledge_state_deltas</code></td>
<td>Who learns what, certainty, how it was learned, who it was shared with</td>
<td><code>knows_edges</code></td>
</tr>
<tr>
<td><span class="badge blue">12</span></td>
<td><code>commitments</code></td>
<td>Foreshadow → payoff pairs (Chekhov's guns), trigger predicates, weight</td>
<td><code>commitments</code></td>
</tr>
<tr>
<td><span class="badge blue">13</span></td>
<td><code>canon_facts</code></td>
<td>Lockable world-facts: kind, subject entity, predicate, value, confidence. Upserts on (novel_id, subject_entity_id, predicate) — skips update if the existing row is locked; a locked fact contradicted by new extraction creates a <code>continuity_flag</code> of type <code>canon_contradiction</code> instead.</td>
<td><code>canon_facts</code>, <code>continuity_flags</code></td>
</tr>
</tbody>
</table>
<div class="callout info">
<strong>Persistence via persist_extras.py, persist_canon.py, and persist_deltas.py:</strong> Passes 9–13 are extracted by <code>ChapterExtractor</code> like all other passes, but their results are written to the DB by dedicated helpers: <code>persist_scenes()</code>, <code>persist_knows_edges()</code>, <code>persist_commitments()</code>, and <code>persist_multi_summaries()</code> in <code>persist_extras.py</code>, and <code>persist_canon_facts()</code> in <code>persist_canon.py</code>. These handle the extra logic needed for those tables (embedding, entity linking, deduplication). <code>persist_canon_facts()</code> upserts on <code>(novel_id, subject_entity_id, predicate)</code>: if the existing row is <strong>locked</strong>, the incoming value is not overwritten — instead a <code>continuity_flag</code> of type <code>canon_contradiction</code> is raised. Pass 3 (<code>state_deltas</code>) is likewise persisted by a dedicated helper, <code>persist_state_deltas()</code> in <code>persist_deltas.py</code>, run inside the same transaction right after <code>persist_canon_facts()</code>.
</div>
<div class="callout tip">
<strong>Custom entities in pass 2:</strong> When a novel has user-defined entity types (e.g., <code>realm</code>, <code>power_system</code> for LitRPG), the <code>new_entities</code> pass prompt is extended with those types and their descriptions. Extracted custom entities land in the <code>custom_entities</code> array in the extraction result and are persisted as <code>entities</code> rows with <code>entity_type = <type_name></code>.
</div>
<h3>Mock Mode</h3>
<p>
Set <code>USE_MOCK_LLM=true</code> (or use <code>--mock</code> CLI flag) to run without an API key.
The mock extractor uses regex + sentence splitting instead of LLM calls.
Embeddings fall back to a deterministic SHA-256 hash. Good for local testing and CI.
</p>
</section>
<!-- ENTITY RESOLUTION -->
<section id="entity-resolution">
<h2>Entity Resolution</h2>
<p>
Entity resolution is the process of mapping an extracted name ("Jane", "Miss Bennet", "Jane B.")
to a stable UUID in the database. This prevents duplicate entity rows for the same person.
</p>
<p>The resolver (<code>resolver.py</code>) uses a 5-step cascade:</p>
<div class="diagram-wrap">
<div class="mermaid">
flowchart TD
A["Extracted name\ne.g. 'Jane B'"] --> B{Exact match\nin DB?}
B -->|yes| Z[Return UUID]
B -->|no| C{Alias match\nin aliases[] array?}
C -->|yes| Z
C -->|no| D{Partial match?\ne.g. 'Jane' matches\n'Jane Bennet'}
D -->|yes| E["Add shorter form as alias\nReturn existing UUID"]
D -->|no| F{Fuzzy match\n≥85% similarity?}
F -->|yes| Z
F -->|no| G["create=True? Create new entity, return new UUID\ncreate=False? Return None (drop reference)"]
style Z fill:#1f6feb,stroke:#58a6ff,color:#e6edf3
style G fill:#3d2b1f,stroke:#f0883e,color:#e6edf3
</div>
</div>
<div class="callout warn">
<strong>Two dedup passes run first</strong> — before entity resolution, the <code>IntraExtractionDeduplicator</code> groups name variants within the extraction (an established identity takes priority; otherwise the model nominates a real input name, preferring a proper name over a description. Conflicting established identities and overlapping generic groups are skipped), then the <code>EntityCanonicalizer</code> merges still-unknown names into existing DB entities (deterministic normalized-equality first, LLM second) — strong-evidence merges write a permanent alias, reasoning-only merges are applied as renames to the current extraction. The resolver therefore sees cleaner input and finds canonicalized names via the renamed extraction or its alias lookup. <code>resolve_any_entity</code> additionally checks every typed table's name <em>and</em> aliases before falling back to creating a character, so an aliased faction/location/object reference can no longer create a phantom character row.
</div>
<div class="callout warn">
<strong>Only <code>new_entities</code> mints characters</strong> — the "Create new entity" leaf above fires only when the caller passes <code>create=True</code>, which just the authoritative <code>new_entities</code> pass does. Every reference-only pass — event <code>involved_characters</code>, scene <code>pov</code>/<code>present_character_names</code>, <code>state_deltas</code>, knowledge <code>learnings</code>, relationship/dynamics/commitment endpoints (<code>resolve_any_entity</code>), and canon-fact subjects — calls with <code>create=False</code> and simply drops the reference when the name resolves to nothing. This is what keeps game-system nouns (an "Archer Class", a "Tutorial" window, a "Skills" screen) out of the <code>characters</code> table: the <code>new_entities</code> prompt already excludes them, and no downstream pass can re-introduce them by naming them as an actor. The <code>new_entities</code> pass runs first within the persist transaction, so a legitimately new character is created before any reference pass needs it.
</div>
<p>
The resolver maintains an <strong>in-memory cache per pipeline run</strong> — so the second time
"Jane Bennet" appears in the same chapter, it skips the DB lookup.
</p>
<h3>Relationship dedupe & repair</h3>
<p>
Persisting a relationship first checks for an existing <em>active</em> edge with the same
directed entity pair and <code>rel_type</code>; a reversed pair matches only when both
assertions are explicitly symmetric. An unchanged active relationship is skipped.
An explicit ending creates a chapter-anchored assertion and links the prior row through
<code>superseded_by_id</code>. Reads apply that link only once the ending's source chapter
is visible. Deleting the ending chapter restores the prior assertion. Different
relationship labels still coexist; semantic equivalence is not inferred.
</p>
<p>
<strong>Repair: entity merge.</strong> When dedup gets it wrong, <code>merge_entities</code>
(<code>pipeline/db/entity_merge.py</code>) merges a duplicate entity into its canonical twin in
one transaction: every reference is repointed (relationships, dynamics, canon facts, commitments,
events arrays, states, scenes, knowledge and possession edges), the source's name and aliases