Skip to content

Commit f153836

Browse files
alvins82claude
andcommitted
feat(log): add append-only activity journal (log.md)
Implements the log.md journal from Karpathy's llm-wiki gist: an append-only record of what happened and when, written by ingest, compile, query, and lint. Each entry is a grep-able heading — `## [YYYY-MM-DDThh:mm:ssZ] operation | description` (ISO 8601 UTC) — followed by a markdown bullet body with page wikilinks and counts. Only headings start with `## [`, so the gist's recipe `grep "^## \[" log.md | tail -5` still returns the most recent operations. Per-operation detail: - ingest: Source (input), Saved (output file), Chars - compile: Sources consumed, Created vs Updated pages, Deleted count - query: cited Pages - lint: error/warning/info counts, Flagged pages Logging lives in the core functions (ingestSource, runCompilePipeline, generateAnswer, lint) so both the CLI and the MCP server are covered. Writes are resilient — a log failure warns but never breaks the operation it records. log.md is gitignored alongside wiki/ and sources/. Token/cost intentionally deferred to a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 176eea5 commit f153836

9 files changed

Lines changed: 394 additions & 2 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ dist/
55
.llm-wiki/
66
/wiki/
77
/sources/
8+
/log.md
89
*.tmp
910
.DS_Store
1011
localdocs/

‎README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,7 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
304304
## Output
305305

306306
```
307+
log.md append-only activity journal (ingests, compiles, queries, lint passes)
307308
wiki/
308309
concepts/ one .md file per concept, with YAML frontmatter
309310
queries/ saved query answers, included in index and retrieval
@@ -316,6 +317,30 @@ wiki/
316317

317318
Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
318319

320+
`log.md` records what happened and when. Each entry is a heading with a fixed
321+
prefix — `## [YYYY-MM-DDThh:mm:ssZ] operation | description` (an ISO 8601 UTC
322+
timestamp) — followed by a short bullet body carrying page wikilinks and counts:
323+
324+
```markdown
325+
## [2026-06-05T09:14:02Z] ingest | Attention Is All You Need
326+
- Source: https://arxiv.org/abs/1706.03762
327+
- Saved: sources/attention-is-all-you-need.md
328+
- Chars: 38,214
329+
330+
## [2026-06-05T09:15:30Z] compile | 1 source(s) → 6 page(s)
331+
- Sources: attention-is-all-you-need.md
332+
- Created: [[self-attention]], [[multi-head-attention]], [[transformer]]
333+
- Updated: [[positional-encoding]]
334+
335+
## [2026-06-05T09:16:11Z] query | What is multi-head attention?
336+
- Pages: [[multi-head-attention]], [[self-attention]]
337+
```
338+
339+
Only headings start with `## [`, so the gist's recipe still works even with the
340+
bodies: `grep "^## \[" log.md | tail -5` shows the five most recent operations.
341+
Where `index.md` organizes content for discovery, `log.md` tracks temporal
342+
progression.
343+
319344
## Local web viewer
320345

321346
Run `llmwiki view` from a project root to browse the compiled wiki in a local browser without Obsidian. The viewer is read-only: it renders `wiki/`, exposes sidebar navigation, search, page metadata, health counts, and provenance/citation chips, but does not mutate sources or generated pages.

‎src/commands/ingest.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import path from "path";
1313
import { readFile } from "fs/promises";
1414
import { buildFrontmatter } from "../utils/markdown.js";
1515
import { saveSource } from "../utils/source-writer.js";
16+
import { appendLog } from "../utils/activity-log.js";
1617
import { MAX_SOURCE_CHARS, MIN_SOURCE_CHARS, SOURCES_DIR, IMAGE_EXTENSIONS, TRANSCRIPT_EXTENSIONS } from "../utils/constants.js";
1718
import * as output from "../utils/output.js";
1819
import ingestWeb from "../ingest/web.js";
@@ -266,6 +267,16 @@ export async function ingestSource(source: string): Promise<IngestResult> {
266267
const document = buildDocument(title, source, result, sourceType);
267268
const savedPath = await saveSource(title, document, source);
268269

270+
// Journal the ingest so log.md mirrors the gist's `ingest | Article Title`
271+
// convention. Covers both CLI and the MCP ingest_source tool.
272+
await appendLog(process.cwd(), "ingest", title, {
273+
details: [
274+
`Source: ${source}`,
275+
`Saved: ${savedPath}`,
276+
`Chars: ${result.content.length.toLocaleString()}`,
277+
],
278+
});
279+
269280
return {
270281
filename: path.basename(savedPath),
271282
charCount: result.content.length,

‎src/commands/query.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ import {
3434
type ChunkEmbeddingEntry,
3535
} from "../utils/embeddings.js";
3636
import { rerankWithBm25 } from "../utils/retrieval.js";
37+
import { appendLog, formatWikilinkList } from "../utils/activity-log.js";
3738
import type { ChunkCitation, QueryResult, RetrievalDebug } from "../utils/types.js";
3839

3940
/** Directories to search when loading selected pages, in priority order. */
@@ -438,6 +439,13 @@ export async function generateAnswer(
438439
const selection = await selectRelevantPages(root, question, Boolean(options.debug));
439440
options.onPageSelection?.(selection.pages, selection.reasoning);
440441

442+
// Journal the question so log.md records queries alongside ingests and
443+
// compiles. Logged here (not in the CLI command) so MCP queries are captured
444+
// too, and after selection so the cited pages can be listed.
445+
await appendLog(root, "query", question, {
446+
details: selection.pages.length > 0 ? [`Pages: ${formatWikilinkList(selection.pages)}`] : [],
447+
});
448+
441449
const pagesContent = await loadSelectedPages(root, selection.pages);
442450

443451
if (!pagesContent) {

‎src/compiler/index.ts‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
* sources are processed through the LLM pipeline.
99
*/
1010

11-
import { readFile } from "fs/promises";
11+
import { readFile, readdir } from "fs/promises";
1212
import path from "path";
1313
import { readState, updateSourceState } from "../utils/state.js";
1414
import {
@@ -48,6 +48,7 @@ import { buildBudgetedCombinedContent, type SourceSlice } from "./prompt-budget.
4848
import { addObsidianMeta, generateMOC } from "./obsidian.js";
4949
import { updateEmbeddings } from "../utils/embeddings.js";
5050
import { writeCandidate } from "./candidates.js";
51+
import { appendLog, formatList, formatWikilinkList } from "../utils/activity-log.js";
5152
import {
5253
checkPageBrokenCitations,
5354
checkPageCrossLinks,
@@ -60,6 +61,7 @@ import {
6061
COMPILE_CONCURRENCY,
6162
CONCEPTS_DIR,
6263
INDEX_FILE,
64+
QUERIES_DIR,
6365
SOURCES_DIR,
6466
} from "../utils/constants.js";
6567
import pLimit from "p-limit";
@@ -195,6 +197,47 @@ async function persistExtractionStates(
195197
}
196198
}
197199

200+
/** List page slugs already on disk (concepts + queries) before a compile run. */
201+
async function listExistingPageSlugs(root: string): Promise<Set<string>> {
202+
const slugs = new Set<string>();
203+
for (const dir of [CONCEPTS_DIR, QUERIES_DIR]) {
204+
try {
205+
const files = await readdir(path.join(root, dir));
206+
for (const file of files) {
207+
if (file.endsWith(".md")) slugs.add(file.slice(0, -3));
208+
}
209+
} catch {
210+
// Directory may not exist yet on a first compile — nothing to snapshot.
211+
}
212+
}
213+
return slugs;
214+
}
215+
216+
/**
217+
* Journal a non-review compile to log.md: the source files consumed, plus the
218+
* produced pages split into created (new on disk) and updated (overwritten).
219+
*/
220+
async function logCompile(
221+
root: string,
222+
buckets: ChangeBuckets,
223+
generation: PageGenerationResult,
224+
existingSlugs: Set<string>,
225+
): Promise<void> {
226+
if (buckets.toCompile.length === 0 && buckets.deleted.length === 0) return;
227+
const produced = [...generation.pages.map((entry) => entry.slug), ...generation.seedSlugs];
228+
const created = produced.filter((slug) => !existingSlugs.has(slug));
229+
const updated = produced.filter((slug) => existingSlugs.has(slug));
230+
const heading = `${buckets.toCompile.length} source(s) → ${produced.length} page(s)`;
231+
const details: string[] = [];
232+
if (buckets.toCompile.length > 0) {
233+
details.push(`Sources: ${formatList(buckets.toCompile.map((change) => change.file))}`);
234+
}
235+
if (created.length > 0) details.push(`Created: ${formatWikilinkList(created)}`);
236+
if (updated.length > 0) details.push(`Updated: ${formatWikilinkList(updated)}`);
237+
if (buckets.deleted.length > 0) details.push(`Deleted sources: ${buckets.deleted.length}`);
238+
await appendLog(root, "compile", heading, { details });
239+
}
240+
198241
/** Build the structured CompileResult and emit the CLI completion banner. */
199242
function summarizeCompile(
200243
buckets: ChangeBuckets,
@@ -301,6 +344,9 @@ async function runCompilePipeline(
301344
await freezeFailedExtractions(root, extractions, frozenSlugs);
302345
}
303346

347+
// Snapshot pages on disk before generation so the journal can tell which
348+
// produced pages are new (created) versus overwritten (updated).
349+
const existingSlugs = await listExistingPageSlugs(root);
304350
const generation = await generatePagesPhase(root, extractions, frozenSlugs, schema, options);
305351

306352
if (!options.review) {
@@ -313,6 +359,7 @@ async function runCompilePipeline(
313359
// to honour the "no wiki/ mutation" contract of that mode.
314360
await generateSeedPages(root, schema, generation);
315361
await finalizeWiki(root, generation.pages, generation.seedSlugs);
362+
await logCompile(root, buckets, generation, existingSlugs);
316363
}
317364
return summarizeCompile(buckets, generation, extractions, options);
318365
}

‎src/linter/index.ts‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,11 @@ import {
2121
checkSchemaCrossLinks,
2222
checkStalePages,
2323
} from "./rules.js";
24+
import path from "path";
2425
import { loadSchema } from "../schema/index.js";
2526
import { buildFreshnessSnapshot } from "../freshness/index.js";
2627
import type { FreshnessSnapshot } from "../freshness/types.js";
28+
import { appendLog, formatWikilinkList } from "../utils/activity-log.js";
2729

2830
/** Rule-only lint checks that don't depend on the schema layer. */
2931
const RULES_WITHOUT_SCHEMA: LintRule[] = [
@@ -73,10 +75,36 @@ export async function lint(root: string): Promise<LintSummary> {
7375

7476
const results = [...plainResults.flat(), ...schemaResults.flat(), ...freshnessResults.flat()];
7577

76-
return {
78+
const summary: LintSummary = {
7779
errors: countBySeverity(results, "error"),
7880
warnings: countBySeverity(results, "warning"),
7981
info: countBySeverity(results, "info"),
8082
results,
8183
};
84+
85+
await logLintPass(root, summary);
86+
return summary;
87+
}
88+
89+
/** Distinct page slugs touched by error/warning diagnostics, in first-seen order. */
90+
function flaggedPageSlugs(results: LintResult[]): string[] {
91+
const slugs: string[] = [];
92+
const seen = new Set<string>();
93+
for (const result of results) {
94+
if (result.severity === "info") continue;
95+
const slug = path.basename(result.file, ".md");
96+
if (seen.has(slug)) continue;
97+
seen.add(slug);
98+
slugs.push(slug);
99+
}
100+
return slugs;
101+
}
102+
103+
/** Journal the lint pass so log.md mirrors the gist's "lint passes" record. */
104+
async function logLintPass(root: string, summary: LintSummary): Promise<void> {
105+
const heading =
106+
`${summary.errors} error(s), ${summary.warnings} warning(s), ${summary.info} info`;
107+
const flagged = flaggedPageSlugs(summary.results);
108+
const details = flagged.length > 0 ? [`Flagged: ${formatWikilinkList(flagged)}`] : [];
109+
await appendLog(root, "lint", heading, { details });
82110
}

‎src/utils/activity-log.ts‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
/**
2+
* Append-only activity log (log.md).
3+
*
4+
* Implements the chronological journal described in Karpathy's llm-wiki gist:
5+
* a single append-only file recording what happened and when — ingests,
6+
* compiles, queries, and lint passes. Each entry is a heading line with a
7+
* fixed prefix — `## [YYYY-MM-DDThh:mm:ssZ] operation | description` —
8+
* optionally followed by a markdown bullet body carrying detail (page
9+
* wikilinks, counts).
10+
*
11+
* The heading prefix is what keeps the log parseable: because only headings
12+
* start with `## [`, the gist's recipe `grep "^## \[" log.md | tail -5` still
13+
* returns the five most recent operations even though entries now have bodies.
14+
* The log complements wiki/index.md: the index organizes content for
15+
* discovery, the log tracks temporal progression.
16+
*/
17+
18+
import { appendFile } from "fs/promises";
19+
import path from "path";
20+
import {
21+
LOG_FILE,
22+
LOG_DESCRIPTION_MAX_CHARS,
23+
LOG_MAX_PAGE_LINKS,
24+
} from "./constants.js";
25+
import * as output from "./output.js";
26+
27+
/** Unicode ellipsis appended when a description is truncated. */
28+
const TRUNCATION_MARKER = "…";
29+
30+
/** Optional extras for a log entry. */
31+
export interface LogEntryOptions {
32+
/** Markdown bullet detail lines rendered under the heading (page links, counts). */
33+
details?: string[];
34+
/** Entry timestamp; rendered as ISO 8601 UTC to second precision. Defaults to now. */
35+
date?: Date;
36+
}
37+
38+
/**
39+
* Collapse whitespace (including newlines) and cap length so the heading stays
40+
* a single, grep-friendly line regardless of what the caller passes in.
41+
*/
42+
function sanitizeDescription(description: string): string {
43+
const singleLine = description.replace(/\s+/g, " ").trim();
44+
if (singleLine.length <= LOG_DESCRIPTION_MAX_CHARS) return singleLine;
45+
return singleLine.slice(0, LOG_DESCRIPTION_MAX_CHARS - 1) + TRUNCATION_MARKER;
46+
}
47+
48+
/**
49+
* Render items as a comma-separated string, truncating past `cap` with a
50+
* "(+N more)" suffix. Returns an empty string for an empty list so callers can
51+
* skip the detail line entirely.
52+
*
53+
* @param items - Pre-rendered list items (filenames, wikilinks, …).
54+
* @param cap - Max items to show before truncating; defaults to the configured limit.
55+
*/
56+
export function formatList(items: string[], cap: number = LOG_MAX_PAGE_LINKS): string {
57+
const shown = items.slice(0, cap);
58+
const remaining = items.length - shown.length;
59+
const suffix = remaining > 0 ? `, … (+${remaining} more)` : "";
60+
return shown.join(", ") + suffix;
61+
}
62+
63+
/**
64+
* Render a list of page slugs as a comma-separated wikilink string, truncating
65+
* past {@link LOG_MAX_PAGE_LINKS}. Returns an empty string for an empty list.
66+
*
67+
* @param slugs - Page slugs to render as `[[slug]]` wikilinks.
68+
* @param cap - Max links to show before truncating; defaults to the configured limit.
69+
*/
70+
export function formatWikilinkList(
71+
slugs: string[],
72+
cap: number = LOG_MAX_PAGE_LINKS,
73+
): string {
74+
return formatList(slugs.map((slug) => `[[${slug}]]`), cap);
75+
}
76+
77+
/**
78+
* Format one log entry: the parseable `## [YYYY-MM-DDThh:mm:ssZ] operation |
79+
* description` heading, plus a `- detail` bullet for each provided detail line.
80+
* Pure and date-injectable so it can be unit-tested deterministically.
81+
*
82+
* @param operation - Short operation tag, e.g. "ingest", "compile", "query".
83+
* @param description - Human-readable heading detail (title, counts, question).
84+
* @param date - The entry's timestamp; rendered as ISO 8601 UTC to second precision.
85+
* @param details - Optional bullet lines rendered under the heading.
86+
*/
87+
export function formatLogEntry(
88+
operation: string,
89+
description: string,
90+
date: Date,
91+
details: string[] = [],
92+
): string {
93+
// ISO 8601 UTC to second precision, e.g. 2026-06-05T14:58:41Z. Sorts
94+
// lexicographically and still matches the gist's `^## \[` grep recipe.
95+
const timestamp = date.toISOString().slice(0, 19) + "Z";
96+
const heading = `## [${timestamp}] ${operation} | ${sanitizeDescription(description)}`;
97+
if (details.length === 0) return heading;
98+
const body = details.map((line) => `- ${line}`).join("\n");
99+
return `${heading}\n${body}`;
100+
}
101+
102+
/**
103+
* Append a single entry to the project's log.md, blank-line separated from the
104+
* previous one.
105+
*
106+
* Resilient by design: the log is a side journal, so a write failure warns but
107+
* never throws — recording an event must not break the operation it records.
108+
*
109+
* @param root - Absolute path to the project root directory.
110+
* @param operation - Short operation tag, e.g. "ingest", "compile", "query".
111+
* @param description - Human-readable heading detail for the entry.
112+
* @param opts - Optional bullet body and entry date.
113+
*/
114+
export async function appendLog(
115+
root: string,
116+
operation: string,
117+
description: string,
118+
opts: LogEntryOptions = {},
119+
): Promise<void> {
120+
try {
121+
const entry = formatLogEntry(operation, description, opts.date ?? new Date(), opts.details);
122+
await appendFile(path.join(root, LOG_FILE), `${entry}\n\n`, "utf-8");
123+
} catch (err) {
124+
const message = err instanceof Error ? err.message : String(err);
125+
output.status("!", output.warn(`Skipped log.md update: ${message}`));
126+
}
127+
}

‎src/utils/constants.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,24 @@ export const STATE_FILE = ".llmwiki/state.json";
8080
export const LOCK_FILE = ".llmwiki/lock";
8181
export const INDEX_FILE = "wiki/index.md";
8282
export const MOC_FILE = "wiki/MOC.md";
83+
84+
/**
85+
* Append-only activity journal at the project root, per Karpathy's llm-wiki
86+
* gist: a chronological record of what happened and when (ingests, compiles,
87+
* queries, lint passes). Lives at the root rather than under wiki/ because it
88+
* spans project-level operations — ingest writes before any wiki/ exists.
89+
*/
90+
export const LOG_FILE = "log.md";
91+
92+
/** Max characters for a single log.md entry description before truncation. */
93+
export const LOG_DESCRIPTION_MAX_CHARS = 200;
94+
95+
/**
96+
* Max number of page wikilinks listed in a single log.md entry detail line.
97+
* Beyond this the list is truncated with a "(+N more)" suffix so a large
98+
* compile doesn't write a wall of links into the journal.
99+
*/
100+
export const LOG_MAX_PAGE_LINKS = 20;
83101
export const EMBEDDINGS_FILE = ".llmwiki/embeddings.json";
84102
export const LAST_LINT_FILE = ".llmwiki/last-lint.json";
85103

0 commit comments

Comments
 (0)