Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ Use them independently or together. The [`@atomicmemory/llmwiki`](https://github

## Contributing

Contributions are welcome. If llmwiki is missing something you need, open an issue or PR and describe the workflow you are trying to support need-driven improvements are often the best ones. If you want to contribute more generally, roadmap items are a good place to start. For larger changes to core compile, review, import/export, or retrieval semantics, please start with an issue or design discussion so we can align on the contract first.
Contributions are welcome. If llmwiki is missing something you need, open an issue or PR and describe the workflow you are trying to support - need-driven improvements are often the best ones. If you want to contribute more generally, roadmap items are a good place to start. For larger changes to core compile, review, import/export, or retrieval semantics, please start with an issue or design discussion so we can align on the contract first.

Before committing code changes, run:

Expand Down
2 changes: 1 addition & 1 deletion docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
## Style preferences

- Use active voice and second person ("you")
- Keep sentences concise one idea per sentence
- Keep sentences concise - one idea per sentence
- Use sentence case for headings
- Bold for UI elements: Click **Settings**
- Code formatting for file names, commands, paths, and code references
Expand Down
14 changes: 7 additions & 7 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ Internal refactors do not need docs unless they change an observable contract.

## Where changes belong

- `introduction.mdx` and `quickstart.mdx` only for first-run or top-level positioning changes.
- `cli/*.mdx` command syntax, flags, examples, output semantics, and command-specific safety notes.
- `configuration/*.mdx` environment variables, providers, project config, schema, review policy, and defaults.
- `concepts/*.mdx` durable concepts such as the wiki model, page types, citations, freshness, provenance, and review lifecycle.
- `guides/*.mdx` end-to-end workflows and integrations.
- `troubleshooting/*.mdx` common failure modes, recovery steps, and diagnostics.
- `docs.json` navigation only. Add new pages here or they are effectively unpublished.
- `introduction.mdx` and `quickstart.mdx` - only for first-run or top-level positioning changes.
- `cli/*.mdx` - command syntax, flags, examples, output semantics, and command-specific safety notes.
- `configuration/*.mdx` - environment variables, providers, project config, schema, review policy, and defaults.
- `concepts/*.mdx` - durable concepts such as the wiki model, page types, citations, freshness, provenance, and review lifecycle.
- `guides/*.mdx` - end-to-end workflows and integrations.
- `troubleshooting/*.mdx` - common failure modes, recovery steps, and diagnostics.
- `docs.json` - navigation only. Add new pages here or they are effectively unpublished.

## Feature PR checklist

Expand Down
12 changes: 6 additions & 6 deletions docs/cli/compile.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "llmwiki compile Generate Wiki Pages from Sources"
title: "llmwiki compile - Generate Wiki Pages from Sources"
sidebarTitle: "Compile"
description: "llmwiki compile runs the incremental two-phase LLM pipeline to extract concepts and generate wiki pages. Key flags: --review and --lang."
---

Compiling is the step that turns raw source files into a structured, interlinked wiki. When you run `llmwiki compile`, the pipeline runs in two phases: **Phase 1** reads every changed source in `sources/` and asks the LLM to extract all concepts, entities, and topics it finds. **Phase 2** takes those extracted concepts and generates typed wiki pages one Markdown file per concept, with YAML frontmatter, paragraph-level citations back to source line ranges, and `[[wikilinks]]` connecting related concepts.
Compiling is the step that turns raw source files into a structured, interlinked wiki. When you run `llmwiki compile`, the pipeline runs in two phases: **Phase 1** reads every changed source in `sources/` and asks the LLM to extract all concepts, entities, and topics it finds. **Phase 2** takes those extracted concepts and generates typed wiki pages - one Markdown file per concept, with YAML frontmatter, paragraph-level citations back to source line ranges, and `[[wikilinks]]` connecting related concepts.

Splitting into two phases matters: by completing all extraction before any pages are written, the compiler can merge concepts that appear across multiple sources into a single page, catch extraction failures before anything is committed to disk, and resolve cross-references that would be impossible to wire up in a single pass.

Expand All @@ -25,7 +25,7 @@ Run from your project root. If `sources/` is empty or doesn't exist yet, compile

## Incremental behaviour

Compile is hash-based and incremental. Each source file's content is fingerprinted on ingest; on subsequent compiles, only sources whose content hash has changed are re-processed through the LLM. Sources that haven't changed are skipped entirely no API calls, no rewrites. This means re-running `llmwiki compile` after editing a single source touches only the pages that source contributed to, even in a large wiki.
Compile is hash-based and incremental. Each source file's content is fingerprinted on ingest; on subsequent compiles, only sources whose content hash has changed are re-processed through the LLM. Sources that haven't changed are skipped entirely - no API calls, no rewrites. This means re-running `llmwiki compile` after editing a single source touches only the pages that source contributed to, even in a large wiki.

The same incremental logic applies to embeddings: chunk-level embeddings are updated only for pages whose underlying content changed.

Expand Down Expand Up @@ -68,15 +68,15 @@ If sources have changed since the last compile, affected pages are marked `stale
llmwiki refresh --stale
```

This recompiles only the sources that own stale pages and cleans up pages whose sources were all deleted (orphaned pages). Unrelated new sources that haven't been compiled yet are deliberately skipped use `llmwiki compile` to bring those in separately.
This recompiles only the sources that own stale pages and cleans up pages whose sources were all deleted (orphaned pages). Unrelated new sources that haven't been compiled yet are deliberately skipped - use `llmwiki compile` to bring those in separately.

### `--dry-run`

```bash
llmwiki refresh --stale --dry-run
```

Prints the plan which pages would be recompiled, which sources are involved, how many orphaned pages would be cleaned up without making any LLM calls or writing any files. Use this to verify the plan before committing.
Prints the plan - which pages would be recompiled, which sources are involved, how many orphaned pages would be cleaned up - without making any LLM calls or writing any files. Use this to verify the plan before committing.

## `llmwiki watch`

Expand All @@ -100,7 +100,7 @@ llmwiki review approve <id>
llmwiki review reject <id>
```

**Review policy (selective):** Add a `.llmwiki/config.json` to hold only pages that trip specific risk conditions low confidence, contradictions, schema violations, or broken provenance while writing the rest live. See [Review Policy](/configuration/review-policy) for the full configuration reference.
**Review policy (selective):** Add a `.llmwiki/config.json` to hold only pages that trip specific risk conditions - low confidence, contradictions, schema violations, or broken provenance - while writing the rest live. See [Review Policy](/configuration/review-policy) for the full configuration reference.

<Note>
`llmwiki compile` sends source content to your configured LLM provider. Make sure your provider credentials are set before running compile. For Anthropic, set `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`. For other providers, see the [Configuration](/configuration) reference.
Expand Down
18 changes: 9 additions & 9 deletions docs/cli/export.mdx
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: "llmwiki export Export Your Wiki to Portable Formats"
title: "llmwiki export - Export Your Wiki to Portable Formats"
sidebarTitle: "Export"
description: "llmwiki export converts your compiled wiki to llms.txt, JSON, JSON-LD, GraphML, Marp slides, or an Open Knowledge Format bundle."
---

Your compiled wiki lives on disk as markdown files, but the knowledge inside it is useful beyond the local viewer and CLI. `llmwiki export` transforms your wiki into portable formats a compact llms.txt file that fits in an LLM context window, a structured JSON envelope for programmatic consumers, a JSON-LD graph for semantic tooling, a GraphML file for graph visualization, a Marp slide deck for presentations, or an Open Knowledge Format bundle for exchange with other knowledge tools.
Your compiled wiki lives on disk as markdown files, but the knowledge inside it is useful beyond the local viewer and CLI. `llmwiki export` transforms your wiki into portable formats - a compact llms.txt file that fits in an LLM context window, a structured JSON envelope for programmatic consumers, a JSON-LD graph for semantic tooling, a GraphML file for graph visualization, a Marp slide deck for presentations, or an Open Knowledge Format bundle for exchange with other knowledge tools.

Export is a pure transformation of existing wiki content. It makes no LLM calls and doesn't modify any files in `wiki/` or `sources/` only writing artifacts under `dist/exports/` or the `--out` directory you choose for directory-style targets.
Export is a pure transformation of existing wiki content. It makes no LLM calls and doesn't modify any files in `wiki/` or `sources/` - only writing artifacts under `dist/exports/` or the `--out` directory you choose for directory-style targets.

---

Expand All @@ -27,12 +27,12 @@ When you omit `--target`, the single-file formats are exported in one pass. The

| Target | Flag value | Output file | What it produces |
|--------|-----------|-------------|-----------------|
| llms.txt | `llms-txt` | `dist/exports/llms.txt` | Concise index per the [llmstxt.org](https://llmstxt.org) spec page titles, summaries, and links. Compact enough to paste into an LLM context window. |
| llms-full.txt | `llms-full-txt` | `dist/exports/llms-full.txt` | Full content version of llms.txt every page's complete body included. Larger but self-contained. |
| llms.txt | `llms-txt` | `dist/exports/llms.txt` | Concise index per the [llmstxt.org](https://llmstxt.org) spec - page titles, summaries, and links. Compact enough to paste into an LLM context window. |
| llms-full.txt | `llms-full-txt` | `dist/exports/llms-full.txt` | Full content version of llms.txt - every page's complete body included. Larger but self-contained. |
| JSON | `json` | `dist/exports/wiki.json` | Structured JSON envelope with per-page metadata (kind, confidence, provenance, citations, aliases, freshness). See [JSON export structure](#json-export-structure) below. |
| JSON-LD | `json-ld` | `dist/exports/wiki.jsonld` | Schema.org JSON-LD graph for semantic web and knowledge graph tooling. |
| GraphML | `graphml` | `dist/exports/wiki.graphml` | Directed wikilink graph as XML. Import into [Gephi](https://gephi.org), Cytoscape, or any GraphML-compatible tool. |
| Marp | `marp` | `dist/exports/wiki.md` | [Marp](https://marp.app) presentation slide deck one slide per concept page. |
| Marp | `marp` | `dist/exports/wiki.md` | [Marp](https://marp.app) presentation slide deck - one slide per concept page. |
| Open Knowledge Format | `okf` | `dist/exports/okf/` | Directory bundle with `index.md`, per-page OKF docs, copied references, and `log.md`. Use this when another tool expects OKF-style markdown bundles. |

### Examples
Expand Down Expand Up @@ -70,16 +70,16 @@ llmwiki export

## `--project-id <id>`

The `--project-id` flag embeds a stable identifier in the JSON export envelope. Downstream importers use this ID to derive deterministic external IDs for each page so if you re-export and re-import, pages map to the same records rather than creating duplicates.
The `--project-id` flag embeds a stable identifier in the JSON export envelope. Downstream importers use this ID to derive deterministic external IDs for each page - so if you re-export and re-import, pages map to the same records rather than creating duplicates.

```bash
llmwiki export --target json --project-id my-research-wiki
```

Valid project IDs match the pattern `/^[a-z0-9][a-z0-9-]{0,62}$/` lowercase letters, digits, and hyphens, starting with a letter or digit, up to 63 characters. The export is aborted before writing any files if the ID is invalid.
Valid project IDs match the pattern `/^[a-z0-9][a-z0-9-]{0,62}$/` - lowercase letters, digits, and hyphens, starting with a letter or digit, up to 63 characters. The export is aborted before writing any files if the ID is invalid.

<Note>
If you're using the [Atomic Memory bridge](/guides/atomic-memory-bridge), always pass the same `--project-id` value on every export. The bridge derives stable memory record IDs from the project ID and page path changing the ID will create duplicate records on re-import.
If you're using the [Atomic Memory bridge](/guides/atomic-memory-bridge), always pass the same `--project-id` value on every export. The bridge derives stable memory record IDs from the project ID and page path - changing the ID will create duplicate records on re-import.
</Note>

---
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/import.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "llmwiki import Import Open Knowledge Format Bundles"
title: "llmwiki import - Import Open Knowledge Format Bundles"
sidebarTitle: "Import"
description: "llmwiki import --okf reads an Open Knowledge Format bundle and stages its documents for review by default. Use --trusted only for bundles you already trust."
---
Expand Down
12 changes: 6 additions & 6 deletions docs/cli/ingest.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "llmwiki ingest Add Sources to Your Wiki Project"
title: "llmwiki ingest - Add Sources to Your Wiki Project"
sidebarTitle: "Ingest"
description: "llmwiki ingest fetches a URL or copies a local file into sources/. Also covers ingest-session for AI session exports and quickstart for one-step setup."
---

Before you can compile a wiki, you need raw material. Ingesting a source means pulling external content a web page, a PDF, a YouTube video transcript, a local Markdown file into your project's `sources/` directory, where it becomes available for the compile pipeline. Every ingest produces a single Markdown file with YAML frontmatter recording the source URL or path, the detected source type, the ingest timestamp, and a truncation flag if the content exceeded the character limit. That file is the stable, content-addressed record the compiler reads.
Before you can compile a wiki, you need raw material. Ingesting a source means pulling external content - a web page, a PDF, a YouTube video transcript, a local Markdown file - into your project's `sources/` directory, where it becomes available for the compile pipeline. Every ingest produces a single Markdown file with YAML frontmatter recording the source URL or path, the detected source type, the ingest timestamp, and a truncation flag if the content exceeded the character limit. That file is the stable, content-addressed record the compiler reads.

You typically ingest first and compile second. If you're just getting started and want to do both in a single step, see [`llmwiki quickstart`](#llmwiki-quickstart) below.

Expand Down Expand Up @@ -61,7 +61,7 @@ llmwiki ingest ./diagrams/system-architecture.png

### SSRF considerations

`llmwiki ingest` is a server-side fetch primitive it follows the URL you give it and reads files from disk. **Only pass trusted input to `llmwiki ingest`.** If you're building a tool that ingests user-supplied URLs or untrusted strings, use the SDK's `ingestText()` method instead. `ingestText` accepts raw text without making any network requests or filesystem reads, so it is safe for untrusted content.
`llmwiki ingest` is a server-side fetch primitive - it follows the URL you give it and reads files from disk. **Only pass trusted input to `llmwiki ingest`.** If you're building a tool that ingests user-supplied URLs or untrusted strings, use the SDK's `ingestText()` method instead. `ingestText` accepts raw text without making any network requests or filesystem reads, so it is safe for untrusted content.

---

Expand All @@ -88,7 +88,7 @@ llmwiki ingest-session ./exports/
llmwiki compile
```

Session files land in `sources/` just like any other ingest result. The compile pipeline treats them identically to web or file sources concepts are extracted from the conversation text and turned into wiki pages.
Session files land in `sources/` just like any other ingest result. The compile pipeline treats them identically to web or file sources - concepts are extracted from the conversation text and turned into wiki pages.

---

Expand All @@ -98,7 +98,7 @@ Session files land in `sources/` just like any other ingest result. The compile
llmwiki quickstart <source>
```

`quickstart` is a first-run convenience wrapper that ingests a source, compiles the wiki, and opens the local viewer all in one command. It is the fastest way to go from zero to a browsable wiki.
`quickstart` is a first-run convenience wrapper that ingests a source, compiles the wiki, and opens the local viewer - all in one command. It is the fastest way to go from zero to a browsable wiki.

Under the hood, `quickstart` runs `ingest` then `compile`. Compile requires LLM credentials; if credentials are missing, the source is still saved to disk before quickstart reports the compile failure, so your content is preserved and you can run `llmwiki compile` manually once credentials are configured.

Expand Down Expand Up @@ -132,5 +132,5 @@ llmwiki quickstart ./brief.md --json
```

<Tip>
After ingesting sources, run `llmwiki next` to get a recommended next action tailored to your project's current state it reads your `sources/`, `wiki/`, and candidate queue to suggest whether you should compile, review candidates, run lint, or query.
After ingesting sources, run `llmwiki next` to get a recommended next action tailored to your project's current state - it reads your `sources/`, `wiki/`, and candidate queue to suggest whether you should compile, review candidates, run lint, or query.
</Tip>
Loading
Loading