Skip to content

Wiki Generation Pipeline

github-actions[bot] edited this page Mar 14, 2026 · 2 revisions

Wiki Generation Pipeline and Claude Integration

This page explains the wiki generation pipeline that orchestrates Claude Code for wiki structure determination and content generation. The pipeline uses a two-stage prompt system: first determining the required documentation structure via XML-based prompts, then generating individual page content using Claude's code analysis capabilities. This architecture replaces traditional RAG/embedding approaches with direct code analysis, allowing Claude to read, search, and analyze repositories in real time while generating documentation.

Overview of the Two-Stage Pipeline

The wiki generation pipeline operates in two distinct stages: structure determination and content generation. This separation of concerns allows the system to first understand the scope and organization of documentation needed, then focus computational resources on generating individual pages in parallel.

flowchart TD
    Start["Start: Repository Input"] --> Validate["Validate Repository URLs"]
    Validate --> Clone["Clone Repository<br/>or Use Local Dir"]
    Clone --> Structure["Stage 1: Structure<br/>Determine Wiki Pages"]
    Structure --> XML["XML Response<br/>Parsed for Page List"]
    XML --> Home["Generate Home.md<br/>& _Sidebar.md"]
    Home --> Parallel["Stage 2: Content<br/>Generate Pages in Parallel"]
    Parallel --> Generate["Generate Page<br/>with Full Code Access"]
    Generate --> Retry["Retry Failed Pages<br/>up to 3 attempts"]
    Retry --> Output["Write .md Files<br/>& _errors.log"]
    Output --> End["Completed Wiki<br/>in Output Directory"]
    style Structure fill:#e1f5ff
    style Parallel fill:#f3e5f5
    style Generate fill:#f3e5f5
Loading

Stage 1: Structure Determination

Purpose and Scope

The structure determination stage analyzes the target repository and decides what documentation pages are needed. Rather than using predefined templates, Claude examines the actual codebase to identify components, patterns, and subsystems that merit documentation. This stage outputs an XML document defining the page structure that will be generated in Stage 2.

Sources: main.go:183-273, main.go:524-546

Structure Prompt System

The structure determination uses a specialized system prompt and user prompt to ensure correct XML output format and thorough code analysis:

System Prompt (lines 175-181):

const xmlSystemPrompt = `CRITICAL INSTRUCTIONS FOR XML RESPONSES:
When the user requests XML output (e.g., wiki_structure, or any XML format):
1. Return ONLY the raw XML - no markdown code fences, no backticks, no explanation
2. Do NOT wrap XML in triple backticks or markdown code blocks
3. Do NOT add any text before or after the XML
4. Start directly with the opening XML tag and end with the closing XML tag
5. Ensure the XML is well-formed and valid`

The system prompt is critical because it prevents Claude from wrapping XML in markdown code blocks, which would make parsing impossible. This explicit instruction is necessary because Claude's default behavior is to wrap code in fences.

User Prompt (lines 183-273): The structurePrompt() function generates a comprehensive analysis prompt that:

  • Provides the project name and repository list
  • Requests direct code analysis using Read, Grep, Glob, and Bash tools
  • Lists documentation categories (Core Documentation, Inferred Documentation)
  • Emphasizes "do NOT create pages for things that don't exist in the repositories"
  • Specifies XML output format with page definitions

Example categories from the prompt (lines 201-230):

### A. Core Documentation (from codefactual)
- **System Overview**: Project purpose, tech stack, directory structure
- **Architecture**: Overall system design, component relationships, design patterns
- **API Specification**: REST/GraphQL endpoints, request/response schemas, authentication
- **Data Model**: Database schema, migrations, ORM models, ER diagrams
- **Configuration & Environment**: Config files, environment variables, feature flags
...
### B. Inferred Documentation (from code patternshigh confidence)
- **Processing Flows**: Key business logic flows derived from function call chains
- **Security Design**: Security measures found in middleware, validation, sanitization
- **Performance Considerations**: Caching, lazy loading, optimization patterns found in code

XML Output Format

The structure prompt requests XML output in a specific format:

<wiki_structure>
  <title>[Overall wiki title]</title>
  <description>[Project description]</description>
  <pages>
    <page id="page-1">
      <title>[Page title]</title>
      <filename>[Page-Filename]</filename>
      <description>[What this page covers and WHY it's needed]</description>
      <importance>high|medium|low</importance>
      <relevant_files>
        <file_path>[Actual file path in the repo]</file_path>
      </relevant_files>
      <related_pages>
        <related>page-2</related>
      </related_pages>
    </page>
  </pages>
</wiki_structure>

Sources: main.go:250-269

XML Parsing and Cleaning

After Claude returns the XML structure, the pipeline cleans and parses it. The cleanXMLResponse() function handles edge cases where Claude might include markdown code fences despite instructions:

func cleanXMLResponse(content string) string {
	content = strings.TrimSpace(content)
	if strings.HasPrefix(content, "```") {
		if idx := strings.Index(content, "\n"); idx != -1 {
			content = content[idx+1:]
		}
		if idx := strings.LastIndex(content, "```"); idx != -1 {
			content = content[:idx]
		}
		content = strings.TrimSpace(content)
	}
	if idx := strings.LastIndex(content, "</wiki_structure>"); idx != -1 {
		content = content[:idx+len("</wiki_structure>")]
	}
	content = strings.ReplaceAll(content, "/no_think", "")
	content = strings.ReplaceAll(content, "/think", "")
	return content
}

This function removes markdown code fences, truncates content after the XML closing tag, and removes Claude-specific directives.

Sources: main.go:380-397

Page Extraction from XML

The parsePages() function extracts page definitions from the cleaned XML:

func parsePages(xml string) []WikiPage {
	var pages []WikiPage
	remaining := xml
	for {
		pageStart := strings.Index(remaining, "<page id=\"")
		if pageStart == -1 {
			break
		}
		remaining = remaining[pageStart:]

		idStart := strings.Index(remaining, "\"") + 1
		idEnd := strings.Index(remaining[idStart:], "\"") + idStart
		id := remaining[idStart:idEnd]

		title := extractTag(remaining, "title")
		filename := extractTag(remaining, "filename")
		desc := extractTag(remaining, "description")

		if filename == "" && title != "" {
			filename = titleToFilename(title)
		}

		if title != "" {
			pages = append(pages, WikiPage{
				ID: id, Title: title, Filename: filename, Description: desc,
			})
		}

		pageEnd := strings.Index(remaining, "</page>")
		if pageEnd == -1 {
			break
		}
		remaining = remaining[pageEnd+7:]
	}
	return pages
}

The parser iterates through the XML, extracting <page> elements and their child tags using extractTag().

Sources: main.go:399-434

Helper Functions for XML Parsing

The extractTag() function provides a generic XML tag extraction utility:

func extractTag(s string, tag string) string {
	open := fmt.Sprintf("<%s>", tag)
	close := fmt.Sprintf("</%s>", tag)
	start := strings.Index(s, open)
	if start == -1 {
		return ""
	}
	start += len(open)
	end := strings.Index(s[start:], close)
	if end == -1 {
		return ""
	}
	return strings.TrimSpace(s[start : start+end])
}

The titleToFilename() function converts page titles to GitHub Wiki-compatible filenames by replacing spaces and special characters with hyphens:

func titleToFilename(title string) string {
	replacer := strings.NewReplacer(
		" ", "-", "/", "-", "\\", "-", ":", "-", "*", "", "?", "",
		"\"", "", "<", "", ">", "", "|", "", "(", "-", ")", "",
		"・", "-", " ", "-",
	)
	return replacer.Replace(title)
}

Sources: main.go:436-458

Stage 2: Content Generation

Overview

After structure determination, the pipeline generates individual page content. Each page is generated in parallel with access to the full repository codebase, allowing Claude to write detailed, well-researched documentation based on actual code analysis.

Page Prompt System

The pagePrompt() function generates a comprehensive prompt for each page, providing:

  • Project name and repository list
  • The specific page to generate (title, description)
  • Full list of other pages for cross-linking
  • Detailed content requirements and formatting guidelines
  • Language specification

Sources: main.go:275-362

Example structure of the page prompt (lines 286-361):

return fmt.Sprintf(`You are an expert technical writer creating a wiki page.

Project: %s
Repositories: %s

You have full access to ALL repository source code via the tools available to you.
USE the Read, Grep, Glob, and Bash tools to read actual source files before writing.
Do NOT guess or speculate — read the code first, then document what you find.

## Your Task
Write the wiki page: **%s**
Page description: %s

## Output Language
Write ALL content in %s.

## Other Wiki Pages (for cross-linking)
%s

## Content Requirements
### 1. Introduction (1-2 paragraphs)
### 2. Detailed Sections (use ## and ### headings)
### 3. Mermaid Diagrams (EXTENSIVELY use these)
### 4. Tables
### 5. Code Snippets
### 6. Source Citations (CRITICAL)
### 7. Cross-Page Links

## Quality Rules
- Facts from code: always include with source citations
- High-confidence inferences: include
- Pure speculation: NEVER include — omit entirely without mentioning the absence
- Be thorough — this is production-grade documentation
...`, projectName, repoList, page.Title, page.Description, langName, pageList.String(), ...)

This prompt makes Claude an expert technical writer with strict source citation requirements and explicit guidance on the documentation philosophy.

Claude CLI Integration

The claudeCall() function orchestrates communication with the Claude CLI:

func claudeCall(claudePath, model string, repoDirs []string, systemPrompt, prompt, workDir string) (string, error) {
	args := []string{"-p", "--output-format", "text", "--dangerously-skip-permissions"}
	if model != "" {
		args = append(args, "--model", model)
	}
	for _, dir := range repoDirs {
		args = append(args, "--add-dir", dir)
	}
	if systemPrompt != "" {
		args = append(args, "--system-prompt", systemPrompt)
	}

	cmd := exec.Command(claudePath, args...)
	cmd.Stdin = strings.NewReader(prompt)
	if workDir != "" {
		cmd.Dir = workDir
	}

	var stdout, stderr bytes.Buffer
	cmd.Stdout = &stdout
	cmd.Stderr = &stderr

	if err := cmd.Run(); err != nil {
		return "", fmt.Errorf("claude: %v\nstderr: %s", err, stderr.String())
	}

	return strings.TrimSpace(stdout.String()), nil
}

Key aspects:

  • Invokes the claude CLI binary with --dangerously-skip-permissions for automation
  • Adds all repository directories with --add-dir flags for full code access
  • Sets working directory to the output wiki directory for file writes
  • Captures both stdout and stderr for error reporting

Sources: main.go:116-143

Page Generation Data Flow

sequenceDiagram
    participant Pipeline as Wiki Pipeline
    participant CLI as Claude CLI
    participant FS as File System

    Pipeline->>Pipeline: Build page prompt<br/>with page details
    Pipeline->>CLI: Invoke claude -p<br/>--add-dir <repos>
    CLI->>FS: Read source files<br/>using tools
    CLI->>FS: Search with Grep/Glob<br/>Analyze with Bash
    FS-->>CLI: File content
    CLI->>CLI: Generate markdown<br/>with source citations
    CLI->>FS: Write .md file<br/>to wiki directory
    FS-->>CLI: File written
    CLI-->>Pipeline: Return stdout
    Pipeline->>Pipeline: Check file exists<br/>and size > 100 bytes
    alt Success
        Pipeline->>Pipeline: Mark page as ok
    else Failure
        Pipeline->>Pipeline: Retry (up to 3 attempts)
    end
Loading

Parallel Page Generation

Pages are generated in parallel using a semaphore pattern to control concurrency:

var pageDone int32
pageSem := make(chan struct{}, pageParallel)
var pageWg sync.WaitGroup

for i := range allPages {
	pageWg.Add(1)
	pageSem <- struct{}{}  // Acquire semaphore
	go func(idx int) {
		defer pageWg.Done()
		defer func() { <-pageSem }()  // Release semaphore

		page := &allPages[idx]
		filename := filepath.Join(wikiDir, page.Filename+".md")
		maxRetries := 3
		var success bool

		for attempt := 1; attempt <= maxRetries; attempt++ {
			os.Remove(filename)
			_, err := claudeCall(claudePath, model, repoDirs, "",
				pagePrompt(*page, allPages, projectName, repos, language), wikiDir)
			if err != nil {
				continue
			}

			written, readErr := os.ReadFile(filename)
			if readErr == nil && len(written) > 100 {
				page.Content = string(written)
				success = true
				break
			}
		}

		if !success {
			appendError(wikiDir, fmt.Sprintf("Page %d/%d: %s — failed after %d attempts",
				idx+1, len(allPages), page.Title, maxRetries))
		}

		atomic.AddInt32(&pageDone, 1)
	}(i)
}
pageWg.Wait()

Sources: main.go:564-615

Retry Mechanism

Each page is automatically retried up to 3 times if it fails. Failure is detected by:

  1. Claude CLI returning an error
  2. Output file not being created
  3. Output file being too small (< 100 bytes)

Failed pages are also logged to _errors.log with timestamp:

func appendError(dir, msg string) {
	errFile := filepath.Join(dir, "_errors.log")
	f, err := os.OpenFile(errFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
	if err != nil {
		return
	}
	defer f.Close()
	fmt.Fprintf(f, "[%s] %s\n", time.Now().Format("15:04:05"), msg)
}

Sources: main.go:462-470, main.go:607-610

Repository Cloning and Code Access

Before either stage begins, the pipeline obtains the target repository code. The gitClone() function provides flexible authentication:

func gitClone(repoURL, token, destDir string) error {
	if _, err := os.Stat(filepath.Join(destDir, ".git")); err == nil {
		// Repository already exists, update it
		cmd := exec.Command("git", "-C", destDir, "pull", "--ff-only")
		cmd.Stdout = os.Stderr
		cmd.Stderr = os.Stderr
		return cmd.Run()
	}

	cloneURL := repoURL
	if token != "" {
		// PAT specified — use HTTPS with token
		cloneURL = strings.Replace(repoURL, "https://",
			fmt.Sprintf("https://%s@", token), 1)
	} else {
		// No PAT — use SSH
		cloneURL = strings.Replace(repoURL, "https://github.com/",
			"git@github.com:", 1)
		if !strings.HasSuffix(cloneURL, ".git") {
			cloneURL += ".git"
		}
	}

	cmd := exec.Command("git", "clone", "--depth=1", "--single-branch",
		cloneURL, destDir)
	cmd.Stdout = os.Stderr
	cmd.Stderr = os.Stderr
	return cmd.Run()
}

Sources: main.go:147-171

Local Directory Option

The pipeline also supports skipping the clone step to use an existing local directory:

if localDir != "" {
	// Use local directory directly (no clone needed)
	absLocal, _ := filepath.Abs(localDir)
	repoDirs = append(repoDirs, absLocal)
} else {
	// ... standard clone process
}

Sources: main.go:490-514

Multi-Repository Projects

For projects with multiple repositories, the pipeline creates a cross-repository wiki by:

  1. Cloning all repositories to the same clone directory
  2. Passing all repository directories to Claude via --add-dir for each
  3. Allowing Claude to analyze cross-repository interactions
  4. Grouping pages under a single project wiki

The structure prompt supports multi-repository rules (lines 225-230):

## Rules for Multi-Repository Projects
- When multiple repositories form one project, create CROSS-REPOSITORY documentation
- Show how repositories interact with each other (e.g., frontend calls backend API)
- Create architecture pages that span all repositories
- Individual repository details should still get their own focused pages

Sources: main.go:495-515, main.go:225-230

Home Page and Sidebar Generation

After structure determination, the pipeline immediately generates the home page and sidebar index:

func writeHomeAndSidebar(wikiDir, projectName, structureContent string,
	allPages []WikiPage, repos []string) {
	var home strings.Builder
	home.WriteString(fmt.Sprintf("# %s\n\n", projectName))
	desc := extractTag(structureContent, "description")
	if desc != "" {
		home.WriteString(fmt.Sprintf("%s\n\n", desc))
	}
	// ... add repository list for multi-repo projects ...
	home.WriteString(fmt.Sprintf("Generated: %s\n\n",
		time.Now().Format("2006-01-02 15:04:05")))
	home.WriteString("## Pages\n\n")
	for _, page := range allPages {
		home.WriteString(fmt.Sprintf("- [%s](%s) — %s\n",
			page.Title, page.Filename, page.Description))
	}
	os.WriteFile(filepath.Join(wikiDir, "Home.md"), []byte(home.String()), 0644)

	// Generate sidebar with navigation structure
	var sidebar strings.Builder
	sidebar.WriteString("**[Home](Home)**\n\n---\n\n")
	for _, page := range allPages {
		sidebar.WriteString(fmt.Sprintf("- [%s](%s)\n",
			page.Title, page.Filename))
	}
	os.WriteFile(filepath.Join(wikiDir, "_Sidebar.md"),
		[]byte(sidebar.String()), 0644)
}

These files provide GitHub Wiki navigation structure and are generated immediately after structure determination, before page generation begins.

Sources: main.go:638-665

Error Handling and Recovery

The pipeline implements a comprehensive error handling strategy:

  1. Input Validation: Repository URLs are validated against a pattern and checked for path traversal:
var validRepoPattern = regexp.MustCompile(`^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`)

func validateRepo(repo string) error {
	if !validRepoPattern.MatchString(repo) {
		return fmt.Errorf("invalid repo format: %q (expected owner/repo)", repo)
	}
	if strings.Contains(repo, "..") {
		return fmt.Errorf("path traversal detected in repo: %q", repo)
	}
	if strings.ContainsAny(repo, ";&|`$(){}[]!~") {
		return fmt.Errorf("invalid characters in repo: %q", repo)
	}
	return nil
}
  1. Error Logging: All failures are logged to _errors.log with timestamps
  2. Retry Logic: Up to 3 automatic retries per page with exponential backoff indication
  3. Graceful Degradation: Pages with failures create stub files indicating generation failure
  4. Dedicated Retry Mode: The -retry flag allows recovering from partial failures

Sources: main.go:78-92, main.go:462-470

Architecture of the Complete Pipeline

graph TD
    Input["Input:<br/>Repos/File/CLI Flags"]
    Validate["Validate Repository<br/>URLs & Paths"]
    Parse["Parse repos.txt<br/>Standalone & Grouped"]
    Clone["Clone All Repos<br/>to Clone Directory"]

    Structure["Stage 1:<br/>Structure Determination"]
    Struct_Prompt["Build Structure Prompt<br/>with Analysis Categories"]
    Struct_Call["Call Claude CLI<br/>with Full Code Access"]
    Struct_XML["Receive XML<br/>Structure Definition"]
    Parse_XML["Clean & Parse XML<br/>Extract Page List"]
    Home["Write Home.md<br/>& _Sidebar.md"]

    Content["Stage 2:<br/>Content Generation"]
    PageLoop["For Each Page<br/>in Parallel"]
    Page_Prompt["Build Page Prompt<br/>with Cross-Links"]
    Page_Call["Call Claude CLI<br/>with Repository Access"]
    File_Write["Write .md File<br/>to Wiki Directory"]
    Retry_Check["Check File Success<br/>& Size"]
    Retry["Retry up to 3x<br/>on Failure"]
    Error_Log["Log Error<br/>to _errors.log"]

    Output["Final Output:<br/>Complete Wiki"]

    Input --> Validate
    Validate --> Parse
    Parse --> Clone
    Clone --> Structure

    Structure --> Struct_Prompt
    Struct_Prompt --> Struct_Call
    Struct_Call --> Struct_XML
    Struct_XML --> Parse_XML
    Parse_XML --> Home

    Home --> Content
    Content --> PageLoop
    PageLoop --> Page_Prompt
    Page_Prompt --> Page_Call
    Page_Call --> File_Write
    File_Write --> Retry_Check

    Retry_Check -->|Success| Output
    Retry_Check -->|Failure| Retry
    Retry -->|Attempts Remain| Page_Call
    Retry -->|Max Retries| Error_Log
    Error_Log --> Output

    style Input fill:#fff3e0
    style Validate fill:#fff3e0
    style Clone fill:#e8f5e9
    style Structure fill:#e1f5ff
    style Struct_Call fill:#e1f5ff
    style Content fill:#f3e5f5
    style Page_Call fill:#f3e5f5
    style Output fill:#e8f5e9
Loading

Data Structures

WikiPage Structure

The WikiPage struct holds page metadata:

type WikiPage struct {
	ID          string  // Unique identifier from XML
	Title       string  // Page title (e.g., "Architecture and Design")
	Filename    string  // GitHub Wiki filename (e.g., "Architecture-and-Design")
	Description string  // One-line page description
	Content     string  // Generated markdown content
}

WikiResult Structure

The WikiResult struct captures generation results for JSON output:

type WikiResult struct {
	Project    string           `json:"project"`
	Repos      []string         `json:"repos"`
	OutputDir  string           `json:"output_dir"`
	Pages      []WikiPageResult `json:"pages"`
	TotalPages int              `json:"total_pages"`
	Failed     int              `json:"failed"`
	Duration   string           `json:"duration"`
	Status     string           `json:"status"`
}

type WikiPageResult struct {
	Title    string `json:"title"`
	Filename string `json:"filename"`
	Size     int    `json:"size"`
	Status   string `json:"status"`  // "ok", "failed", "pending"
}

Sources: main.go:62-112

Parallelism Strategy

The pipeline supports parallelism at two levels:

  1. Repository-Level Parallelism (-p flag): Multiple projects/repositories are processed simultaneously using a semaphore:
sem := make(chan struct{}, parallel)
for _, t := range tasks {
	wg.Add(1)
	sem <- struct{}{}      // Acquire
	go func(t task) {
		defer wg.Done()
		defer func() { <-sem }()  // Release
		// Process task
	}(t)
}
wg.Wait()
  1. Page-Level Parallelism (-pp flag): Within each project, pages are generated in parallel with a separate semaphore.

Both use Go's semaphore pattern with channels and sync.WaitGroup for coordination.

Sources: main.go:1006-1035, main.go:564-615

Progress Tracking

The Progress struct provides real-time progress reporting to stderr:

type Progress struct {
	mu         sync.Mutex
	totalItems int
	doneItems  int32
	current    map[string]string
}

Progress displays percentage completion, current task name, and status:

[5/10 50%] project1 📝 5/8 (62%) Architecture | project2 📥 cloning ...

Sources: main.go:19-58

Generated Output Structure

The pipeline creates the following files in the output directory:

wiki-output/
  <project-name>/
    Home.md              # Project overview and page index
    _Sidebar.md          # GitHub Wiki sidebar navigation
    <page-1>.md          # Generated documentation page
    <page-2>.md
    ...
    _errors.log          # Log of page generation failures (if any)

Each page is a standalone Markdown file with complete documentation, source citations, and cross-links to other pages using GitHub Wiki format: [Page Title](Page-Filename)

Integration with GitHub Actions

The pipeline is designed to work with GitHub Actions workflows. JSON output mode (-json flag) produces structured results for programmatic integration:

[
  {
    "project": "myproject",
    "repos": ["owner/repo"],
    "output_dir": "/path/to/wiki-output/myproject",
    "pages": [
      {
        "title": "Architecture and Design",
        "filename": "Architecture-and-Design",
        "size": 15248,
        "status": "ok"
      }
    ],
    "total_pages": 12,
    "failed": 0,
    "duration": "45s",
    "status": "completed"
  }
]

Sources: main.go:96-112

Related Pages

Clone this wiki locally