-
Notifications
You must be signed in to change notification settings - Fork 1
Wiki Generation Pipeline
This page explains the wiki generation pipeline that orchestrates Claude Code for wiki structure determination and content generation. The pipeline uses a two-stage prompt system: first determining the required documentation structure via XML-based prompts, then generating individual page content using Claude's code analysis capabilities. This architecture replaces traditional RAG/embedding approaches with direct code analysis, allowing Claude to read, search, and analyze repositories in real time while generating documentation.
The wiki generation pipeline operates in two distinct stages: structure determination and content generation. This separation of concerns allows the system to first understand the scope and organization of documentation needed, then focus computational resources on generating individual pages in parallel.
flowchart TD
Start["Start: Repository Input"] --> Validate["Validate Repository URLs"]
Validate --> Clone["Clone Repository<br/>or Use Local Dir"]
Clone --> Structure["Stage 1: Structure<br/>Determine Wiki Pages"]
Structure --> XML["XML Response<br/>Parsed for Page List"]
XML --> Home["Generate Home.md<br/>& _Sidebar.md"]
Home --> Parallel["Stage 2: Content<br/>Generate Pages in Parallel"]
Parallel --> Generate["Generate Page<br/>with Full Code Access"]
Generate --> Retry["Retry Failed Pages<br/>up to 3 attempts"]
Retry --> Output["Write .md Files<br/>& _errors.log"]
Output --> End["Completed Wiki<br/>in Output Directory"]
style Structure fill:#e1f5ff
style Parallel fill:#f3e5f5
style Generate fill:#f3e5f5
The structure determination stage analyzes the target repository and decides what documentation pages are needed. Rather than using predefined templates, Claude examines the actual codebase to identify components, patterns, and subsystems that merit documentation. This stage outputs an XML document defining the page structure that will be generated in Stage 2.
Sources: main.go:183-273, main.go:524-546
The structure determination uses a specialized system prompt and user prompt to ensure correct XML output format and thorough code analysis:
System Prompt (lines 175-181):
const xmlSystemPrompt = `CRITICAL INSTRUCTIONS FOR XML RESPONSES:
When the user requests XML output (e.g., wiki_structure, or any XML format):
1. Return ONLY the raw XML - no markdown code fences, no backticks, no explanation
2. Do NOT wrap XML in triple backticks or markdown code blocks
3. Do NOT add any text before or after the XML
4. Start directly with the opening XML tag and end with the closing XML tag
5. Ensure the XML is well-formed and valid`The system prompt is critical because it prevents Claude from wrapping XML in markdown code blocks, which would make parsing impossible. This explicit instruction is necessary because Claude's default behavior is to wrap code in fences.
User Prompt (lines 183-273):
The structurePrompt() function generates a comprehensive analysis prompt that:
- Provides the project name and repository list
- Requests direct code analysis using Read, Grep, Glob, and Bash tools
- Lists documentation categories (Core Documentation, Inferred Documentation)
- Emphasizes "do NOT create pages for things that don't exist in the repositories"
- Specifies XML output format with page definitions
Example categories from the prompt (lines 201-230):
### A. Core Documentation (from code — factual)
- **System Overview**: Project purpose, tech stack, directory structure
- **Architecture**: Overall system design, component relationships, design patterns
- **API Specification**: REST/GraphQL endpoints, request/response schemas, authentication
- **Data Model**: Database schema, migrations, ORM models, ER diagrams
- **Configuration & Environment**: Config files, environment variables, feature flags
...
### B. Inferred Documentation (from code patterns — high confidence)
- **Processing Flows**: Key business logic flows derived from function call chains
- **Security Design**: Security measures found in middleware, validation, sanitization
- **Performance Considerations**: Caching, lazy loading, optimization patterns found in codeThe structure prompt requests XML output in a specific format:
<wiki_structure>
<title>[Overall wiki title]</title>
<description>[Project description]</description>
<pages>
<page id="page-1">
<title>[Page title]</title>
<filename>[Page-Filename]</filename>
<description>[What this page covers and WHY it's needed]</description>
<importance>high|medium|low</importance>
<relevant_files>
<file_path>[Actual file path in the repo]</file_path>
</relevant_files>
<related_pages>
<related>page-2</related>
</related_pages>
</page>
</pages>
</wiki_structure>Sources: main.go:250-269
After Claude returns the XML structure, the pipeline cleans and parses it. The cleanXMLResponse() function handles edge cases where Claude might include markdown code fences despite instructions:
func cleanXMLResponse(content string) string {
content = strings.TrimSpace(content)
if strings.HasPrefix(content, "```") {
if idx := strings.Index(content, "\n"); idx != -1 {
content = content[idx+1:]
}
if idx := strings.LastIndex(content, "```"); idx != -1 {
content = content[:idx]
}
content = strings.TrimSpace(content)
}
if idx := strings.LastIndex(content, "</wiki_structure>"); idx != -1 {
content = content[:idx+len("</wiki_structure>")]
}
content = strings.ReplaceAll(content, "/no_think", "")
content = strings.ReplaceAll(content, "/think", "")
return content
}This function removes markdown code fences, truncates content after the XML closing tag, and removes Claude-specific directives.
Sources: main.go:380-397
The parsePages() function extracts page definitions from the cleaned XML:
func parsePages(xml string) []WikiPage {
var pages []WikiPage
remaining := xml
for {
pageStart := strings.Index(remaining, "<page id=\"")
if pageStart == -1 {
break
}
remaining = remaining[pageStart:]
idStart := strings.Index(remaining, "\"") + 1
idEnd := strings.Index(remaining[idStart:], "\"") + idStart
id := remaining[idStart:idEnd]
title := extractTag(remaining, "title")
filename := extractTag(remaining, "filename")
desc := extractTag(remaining, "description")
if filename == "" && title != "" {
filename = titleToFilename(title)
}
if title != "" {
pages = append(pages, WikiPage{
ID: id, Title: title, Filename: filename, Description: desc,
})
}
pageEnd := strings.Index(remaining, "</page>")
if pageEnd == -1 {
break
}
remaining = remaining[pageEnd+7:]
}
return pages
}The parser iterates through the XML, extracting <page> elements and their child tags using extractTag().
Sources: main.go:399-434
The extractTag() function provides a generic XML tag extraction utility:
func extractTag(s string, tag string) string {
open := fmt.Sprintf("<%s>", tag)
close := fmt.Sprintf("</%s>", tag)
start := strings.Index(s, open)
if start == -1 {
return ""
}
start += len(open)
end := strings.Index(s[start:], close)
if end == -1 {
return ""
}
return strings.TrimSpace(s[start : start+end])
}The titleToFilename() function converts page titles to GitHub Wiki-compatible filenames by replacing spaces and special characters with hyphens:
func titleToFilename(title string) string {
replacer := strings.NewReplacer(
" ", "-", "/", "-", "\\", "-", ":", "-", "*", "", "?", "",
"\"", "", "<", "", ">", "", "|", "", "(", "-", ")", "",
"・", "-", " ", "-",
)
return replacer.Replace(title)
}Sources: main.go:436-458
After structure determination, the pipeline generates individual page content. Each page is generated in parallel with access to the full repository codebase, allowing Claude to write detailed, well-researched documentation based on actual code analysis.
The pagePrompt() function generates a comprehensive prompt for each page, providing:
- Project name and repository list
- The specific page to generate (title, description)
- Full list of other pages for cross-linking
- Detailed content requirements and formatting guidelines
- Language specification
Sources: main.go:275-362
Example structure of the page prompt (lines 286-361):
return fmt.Sprintf(`You are an expert technical writer creating a wiki page.
Project: %s
Repositories: %s
You have full access to ALL repository source code via the tools available to you.
USE the Read, Grep, Glob, and Bash tools to read actual source files before writing.
Do NOT guess or speculate — read the code first, then document what you find.
## Your Task
Write the wiki page: **%s**
Page description: %s
## Output Language
Write ALL content in %s.
## Other Wiki Pages (for cross-linking)
%s
## Content Requirements
### 1. Introduction (1-2 paragraphs)
### 2. Detailed Sections (use ## and ### headings)
### 3. Mermaid Diagrams (EXTENSIVELY use these)
### 4. Tables
### 5. Code Snippets
### 6. Source Citations (CRITICAL)
### 7. Cross-Page Links
## Quality Rules
- Facts from code: always include with source citations
- High-confidence inferences: include
- Pure speculation: NEVER include — omit entirely without mentioning the absence
- Be thorough — this is production-grade documentation
...`, projectName, repoList, page.Title, page.Description, langName, pageList.String(), ...)This prompt makes Claude an expert technical writer with strict source citation requirements and explicit guidance on the documentation philosophy.
The claudeCall() function orchestrates communication with the Claude CLI:
func claudeCall(claudePath, model string, repoDirs []string, systemPrompt, prompt, workDir string) (string, error) {
args := []string{"-p", "--output-format", "text", "--dangerously-skip-permissions"}
if model != "" {
args = append(args, "--model", model)
}
for _, dir := range repoDirs {
args = append(args, "--add-dir", dir)
}
if systemPrompt != "" {
args = append(args, "--system-prompt", systemPrompt)
}
cmd := exec.Command(claudePath, args...)
cmd.Stdin = strings.NewReader(prompt)
if workDir != "" {
cmd.Dir = workDir
}
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("claude: %v\nstderr: %s", err, stderr.String())
}
return strings.TrimSpace(stdout.String()), nil
}Key aspects:
- Invokes the
claudeCLI binary with--dangerously-skip-permissionsfor automation - Adds all repository directories with
--add-dirflags for full code access - Sets working directory to the output wiki directory for file writes
- Captures both stdout and stderr for error reporting
Sources: main.go:116-143
sequenceDiagram
participant Pipeline as Wiki Pipeline
participant CLI as Claude CLI
participant FS as File System
Pipeline->>Pipeline: Build page prompt<br/>with page details
Pipeline->>CLI: Invoke claude -p<br/>--add-dir <repos>
CLI->>FS: Read source files<br/>using tools
CLI->>FS: Search with Grep/Glob<br/>Analyze with Bash
FS-->>CLI: File content
CLI->>CLI: Generate markdown<br/>with source citations
CLI->>FS: Write .md file<br/>to wiki directory
FS-->>CLI: File written
CLI-->>Pipeline: Return stdout
Pipeline->>Pipeline: Check file exists<br/>and size > 100 bytes
alt Success
Pipeline->>Pipeline: Mark page as ok
else Failure
Pipeline->>Pipeline: Retry (up to 3 attempts)
end
Pages are generated in parallel using a semaphore pattern to control concurrency:
var pageDone int32
pageSem := make(chan struct{}, pageParallel)
var pageWg sync.WaitGroup
for i := range allPages {
pageWg.Add(1)
pageSem <- struct{}{} // Acquire semaphore
go func(idx int) {
defer pageWg.Done()
defer func() { <-pageSem }() // Release semaphore
page := &allPages[idx]
filename := filepath.Join(wikiDir, page.Filename+".md")
maxRetries := 3
var success bool
for attempt := 1; attempt <= maxRetries; attempt++ {
os.Remove(filename)
_, err := claudeCall(claudePath, model, repoDirs, "",
pagePrompt(*page, allPages, projectName, repos, language), wikiDir)
if err != nil {
continue
}
written, readErr := os.ReadFile(filename)
if readErr == nil && len(written) > 100 {
page.Content = string(written)
success = true
break
}
}
if !success {
appendError(wikiDir, fmt.Sprintf("Page %d/%d: %s — failed after %d attempts",
idx+1, len(allPages), page.Title, maxRetries))
}
atomic.AddInt32(&pageDone, 1)
}(i)
}
pageWg.Wait()Sources: main.go:564-615
Each page is automatically retried up to 3 times if it fails. Failure is detected by:
- Claude CLI returning an error
- Output file not being created
- Output file being too small (< 100 bytes)
Failed pages are also logged to _errors.log with timestamp:
func appendError(dir, msg string) {
errFile := filepath.Join(dir, "_errors.log")
f, err := os.OpenFile(errFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err != nil {
return
}
defer f.Close()
fmt.Fprintf(f, "[%s] %s\n", time.Now().Format("15:04:05"), msg)
}Sources: main.go:462-470, main.go:607-610
Before either stage begins, the pipeline obtains the target repository code. The gitClone() function provides flexible authentication:
func gitClone(repoURL, token, destDir string) error {
if _, err := os.Stat(filepath.Join(destDir, ".git")); err == nil {
// Repository already exists, update it
cmd := exec.Command("git", "-C", destDir, "pull", "--ff-only")
cmd.Stdout = os.Stderr
cmd.Stderr = os.Stderr
return cmd.Run()
}
cloneURL := repoURL
if token != "" {
// PAT specified — use HTTPS with token
cloneURL = strings.Replace(repoURL, "https://",
fmt.Sprintf("https://%s@", token), 1)
} else {
// No PAT — use SSH
cloneURL = strings.Replace(repoURL, "https://github.com/",
"git@github.com:", 1)
if !strings.HasSuffix(cloneURL, ".git") {
cloneURL += ".git"
}
}
cmd := exec.Command("git", "clone", "--depth=1", "--single-branch",
cloneURL, destDir)
cmd.Stdout = os.Stderr
cmd.Stderr = os.Stderr
return cmd.Run()
}Sources: main.go:147-171
The pipeline also supports skipping the clone step to use an existing local directory:
if localDir != "" {
// Use local directory directly (no clone needed)
absLocal, _ := filepath.Abs(localDir)
repoDirs = append(repoDirs, absLocal)
} else {
// ... standard clone process
}Sources: main.go:490-514
For projects with multiple repositories, the pipeline creates a cross-repository wiki by:
- Cloning all repositories to the same clone directory
- Passing all repository directories to Claude via
--add-dirfor each - Allowing Claude to analyze cross-repository interactions
- Grouping pages under a single project wiki
The structure prompt supports multi-repository rules (lines 225-230):
## Rules for Multi-Repository Projects
- When multiple repositories form one project, create CROSS-REPOSITORY documentation
- Show how repositories interact with each other (e.g., frontend calls backend API)
- Create architecture pages that span all repositories
- Individual repository details should still get their own focused pages
Sources: main.go:495-515, main.go:225-230
After structure determination, the pipeline immediately generates the home page and sidebar index:
func writeHomeAndSidebar(wikiDir, projectName, structureContent string,
allPages []WikiPage, repos []string) {
var home strings.Builder
home.WriteString(fmt.Sprintf("# %s\n\n", projectName))
desc := extractTag(structureContent, "description")
if desc != "" {
home.WriteString(fmt.Sprintf("%s\n\n", desc))
}
// ... add repository list for multi-repo projects ...
home.WriteString(fmt.Sprintf("Generated: %s\n\n",
time.Now().Format("2006-01-02 15:04:05")))
home.WriteString("## Pages\n\n")
for _, page := range allPages {
home.WriteString(fmt.Sprintf("- [%s](%s) — %s\n",
page.Title, page.Filename, page.Description))
}
os.WriteFile(filepath.Join(wikiDir, "Home.md"), []byte(home.String()), 0644)
// Generate sidebar with navigation structure
var sidebar strings.Builder
sidebar.WriteString("**[Home](Home)**\n\n---\n\n")
for _, page := range allPages {
sidebar.WriteString(fmt.Sprintf("- [%s](%s)\n",
page.Title, page.Filename))
}
os.WriteFile(filepath.Join(wikiDir, "_Sidebar.md"),
[]byte(sidebar.String()), 0644)
}These files provide GitHub Wiki navigation structure and are generated immediately after structure determination, before page generation begins.
Sources: main.go:638-665
The pipeline implements a comprehensive error handling strategy:
- Input Validation: Repository URLs are validated against a pattern and checked for path traversal:
var validRepoPattern = regexp.MustCompile(`^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`)
func validateRepo(repo string) error {
if !validRepoPattern.MatchString(repo) {
return fmt.Errorf("invalid repo format: %q (expected owner/repo)", repo)
}
if strings.Contains(repo, "..") {
return fmt.Errorf("path traversal detected in repo: %q", repo)
}
if strings.ContainsAny(repo, ";&|`$(){}[]!~") {
return fmt.Errorf("invalid characters in repo: %q", repo)
}
return nil
}-
Error Logging: All failures are logged to
_errors.logwith timestamps - Retry Logic: Up to 3 automatic retries per page with exponential backoff indication
- Graceful Degradation: Pages with failures create stub files indicating generation failure
-
Dedicated Retry Mode: The
-retryflag allows recovering from partial failures
Sources: main.go:78-92, main.go:462-470
graph TD
Input["Input:<br/>Repos/File/CLI Flags"]
Validate["Validate Repository<br/>URLs & Paths"]
Parse["Parse repos.txt<br/>Standalone & Grouped"]
Clone["Clone All Repos<br/>to Clone Directory"]
Structure["Stage 1:<br/>Structure Determination"]
Struct_Prompt["Build Structure Prompt<br/>with Analysis Categories"]
Struct_Call["Call Claude CLI<br/>with Full Code Access"]
Struct_XML["Receive XML<br/>Structure Definition"]
Parse_XML["Clean & Parse XML<br/>Extract Page List"]
Home["Write Home.md<br/>& _Sidebar.md"]
Content["Stage 2:<br/>Content Generation"]
PageLoop["For Each Page<br/>in Parallel"]
Page_Prompt["Build Page Prompt<br/>with Cross-Links"]
Page_Call["Call Claude CLI<br/>with Repository Access"]
File_Write["Write .md File<br/>to Wiki Directory"]
Retry_Check["Check File Success<br/>& Size"]
Retry["Retry up to 3x<br/>on Failure"]
Error_Log["Log Error<br/>to _errors.log"]
Output["Final Output:<br/>Complete Wiki"]
Input --> Validate
Validate --> Parse
Parse --> Clone
Clone --> Structure
Structure --> Struct_Prompt
Struct_Prompt --> Struct_Call
Struct_Call --> Struct_XML
Struct_XML --> Parse_XML
Parse_XML --> Home
Home --> Content
Content --> PageLoop
PageLoop --> Page_Prompt
Page_Prompt --> Page_Call
Page_Call --> File_Write
File_Write --> Retry_Check
Retry_Check -->|Success| Output
Retry_Check -->|Failure| Retry
Retry -->|Attempts Remain| Page_Call
Retry -->|Max Retries| Error_Log
Error_Log --> Output
style Input fill:#fff3e0
style Validate fill:#fff3e0
style Clone fill:#e8f5e9
style Structure fill:#e1f5ff
style Struct_Call fill:#e1f5ff
style Content fill:#f3e5f5
style Page_Call fill:#f3e5f5
style Output fill:#e8f5e9
The WikiPage struct holds page metadata:
type WikiPage struct {
ID string // Unique identifier from XML
Title string // Page title (e.g., "Architecture and Design")
Filename string // GitHub Wiki filename (e.g., "Architecture-and-Design")
Description string // One-line page description
Content string // Generated markdown content
}The WikiResult struct captures generation results for JSON output:
type WikiResult struct {
Project string `json:"project"`
Repos []string `json:"repos"`
OutputDir string `json:"output_dir"`
Pages []WikiPageResult `json:"pages"`
TotalPages int `json:"total_pages"`
Failed int `json:"failed"`
Duration string `json:"duration"`
Status string `json:"status"`
}
type WikiPageResult struct {
Title string `json:"title"`
Filename string `json:"filename"`
Size int `json:"size"`
Status string `json:"status"` // "ok", "failed", "pending"
}Sources: main.go:62-112
The pipeline supports parallelism at two levels:
-
Repository-Level Parallelism (
-pflag): Multiple projects/repositories are processed simultaneously using a semaphore:
sem := make(chan struct{}, parallel)
for _, t := range tasks {
wg.Add(1)
sem <- struct{}{} // Acquire
go func(t task) {
defer wg.Done()
defer func() { <-sem }() // Release
// Process task
}(t)
}
wg.Wait()-
Page-Level Parallelism (
-ppflag): Within each project, pages are generated in parallel with a separate semaphore.
Both use Go's semaphore pattern with channels and sync.WaitGroup for coordination.
Sources: main.go:1006-1035, main.go:564-615
The Progress struct provides real-time progress reporting to stderr:
type Progress struct {
mu sync.Mutex
totalItems int
doneItems int32
current map[string]string
}Progress displays percentage completion, current task name, and status:
[5/10 50%] project1 📝 5/8 (62%) Architecture | project2 📥 cloning ...
Sources: main.go:19-58
The pipeline creates the following files in the output directory:
wiki-output/
<project-name>/
Home.md # Project overview and page index
_Sidebar.md # GitHub Wiki sidebar navigation
<page-1>.md # Generated documentation page
<page-2>.md
...
_errors.log # Log of page generation failures (if any)
Each page is a standalone Markdown file with complete documentation, source citations, and cross-links to other pages using GitHub Wiki format: [Page Title](Page-Filename)
The pipeline is designed to work with GitHub Actions workflows. JSON output mode (-json flag) produces structured results for programmatic integration:
[
{
"project": "myproject",
"repos": ["owner/repo"],
"output_dir": "/path/to/wiki-output/myproject",
"pages": [
{
"title": "Architecture and Design",
"filename": "Architecture-and-Design",
"size": 15248,
"status": "ok"
}
],
"total_pages": 12,
"failed": 0,
"duration": "45s",
"status": "completed"
}
]Sources: main.go:96-112
- System Overview — High-level introduction to wikigen, core functionality, and why direct code analysis replaces RAG approaches
- Repository Analysis Process — Details on repository cloning, structure determination, and page generation phases
- Architecture and Design — Detailed system architecture and component relationships
- CLI Reference and Usage — Complete reference for all command-line flags and options
- Error Handling and Recovery — Error handling strategies and recovery mechanisms
- Parallelism and Performance — Explanation of parallel processing and performance optimization
- JSON Output and Integration — JSON output format for programmatic integration
- Multi-Repository and Grouped Wiki Support — Multi-repository wiki generation and cross-repository documentation
- System Overview
- Architecture & Design
- CLI Usage & Commands
- Configuration & Environment
- Input Formats & Repository Configuration
- Authentication & Git Integration
- Output Format & Wiki Structure
- Error Handling & Retry Mechanism
- Parallel Processing & Performance
- Input Validation & Security
- Build & Deployment
- Claude Code Integration
- Wiki Generation Processing Flow
- Multi-Repository Wiki Support
- Progress Tracking & Output Modes